Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

640 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
58 views

Is this a good approach to solving Atari's "Montezuma's Revenge"?

I'm new to Reinforcement Learning. For an internship, I am currently training Atari's "Montezuma's Revenge" using a double Deep Q-Network with Hindsight Experience Replay (HER) (see also ...
3
votes
1answer
147 views

What is the purpose of argmax in the PPO algorithm?

I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website : This line is ...
3
votes
0answers
54 views

How to deal with approximate states when doing path planning?

If one is interested in implementing a path planning algorithm that is grid-based, one needs to consider the fact that your grid points will never represent the true state of the robot. How is this ...
3
votes
0answers
50 views

Evaluation a policy learned using Q - learning

I have been reading literature on reinforcement learning in healthcare. I am slightly confused between the policy evaluation for both SARSA and Q-learning. To my knowledge, I believe that SARSA is ...
3
votes
0answers
57 views

Are there reinforcement learning algorithms not based on Markov decision processes?

Are all RL algorithms based on the MDP? If not, could you give examples of some which aren't? I've looked elsewhere, but I haven't seen it explicitly said.
3
votes
0answers
55 views

How exactly does self-play work, and how does it relate to MCTS?

I am working towards using RL to create an AI for a two-player, hidden-information, a turn-based board game. I have just finished David Silver's RL course and Denny Britz's coding exercises, and so am ...
3
votes
0answers
43 views

How to deal with nonstationary rewards in asymmetric self-play reinforcement learning?

Suppose we're training two agents to play an asymmetric game from scratch using self play (like Zerg vs. Protoss in Starcraft). During training one of the agents can become stronger (discover a good ...
3
votes
0answers
39 views

Reinforcement Learning on quantum circuit

I am trying to teach an agent to make any random 1-qubit state reach uniform superposition. So basically, the full circuit will be ...
3
votes
1answer
124 views

Understanding proof of lemma 1 (policy improvement bound) of the "Trust Region Policy Optimization" paper

In the Trust Region Policy Optimization paper, in Lemma 1 of Appendix A, I did not quite understand the transition from (21) from (20). In going from (20) to (21), $A^\pi(s_t, a_t)$ is substituted ...
3
votes
0answers
51 views

Ideas on a network that can translate image differences into motor commands?

I'd like to design a network that gets two images (an image under construction, and an ideal image), and has to come up with an action vector for a simple motor command which would augment the image ...
3
votes
0answers
152 views

Is there a way to do reinforcement learning in POMDP?

Are there any algorithms to use reinforcement learning to learn optimal policies in partially observable Markov decision process (POMDP) i.e. when the state is not perfectly observed. More ...
3
votes
0answers
83 views

What is the difference between random and sequential sampling from the reply memory?

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure. ...
3
votes
0answers
73 views

Feasibility of using machine learning to obtain self-consistent solutions

I am a physicist and I don't have much background on machine learning or deep learning except taking a couple of courses on statistics. In physics, we often simulate a model by means of two-way ...
3
votes
0answers
29 views

Should importance sample weighting be compensated for by dynamically increasing learning rate?

I'm using Prioritized Experience Replay (PER) with a DDQN. To compensate for overfitting relatively high-value samples due to the non-uniform selection, I'm training with sample weights provided along ...
3
votes
1answer
106 views

How are the observations stored in the RNN that encodes the state?

I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an ...
3
votes
0answers
104 views

What is the meaning of the words 'bias' and 'variance' in RL?

In reinforcement learning approaches, like temporal-difference (TD) learning or Monte Carlo methods, two of the metrics used to measure their performance are the bias and the variance. What do these ...
3
votes
0answers
90 views

How can I use Q-learning for inventory decision making?

I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level of $S$. That is replenishment orders are placed for every demand arrival to ...
3
votes
0answers
65 views

Designing state representation for board game

I am trying to write self-play RL (NN + MCTS http://web.stanford.edu/~surag/posts/alphazero.html) to "solve" a board game. However, I got stuck in designing boardgame same (input layer for NN). 1) ...
3
votes
1answer
496 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
3
votes
0answers
71 views

How to represent action space in reinforcement learning?

I started to learn reinforcement learning a few days ago. And I want to use that to solve resource allocation problem something like given a constant number, find the best way to divide it into ...
3
votes
0answers
32 views

Training a reinforcement learning model with multiple images

I am tentatively trying to train a deep reinforcement learning model the maze escaping task, and each time it takes one image as the input (e.g., a different "maze"). Suppose I have about $10K$ ...
3
votes
0answers
433 views

Implementation of PPO - Value Loss not converging, return plateauing

Copy from my reddit post: (Sorry if this does not fit here, please tell me and i delete it) Help regarding I'm working on an implementation of PPO, which i plan to use in my (Bachelors) Thesis. To ...
3
votes
0answers
160 views

When using hashing in tile coding, why are memory requirements reduced and there is only a little loss of performance?

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto explain, on page 221, a form of tile coding using hashing, to reduce memory consumption. I have two questions ...
3
votes
0answers
38 views

Can $\Phi$ measure of Integrated Information Theory serve as reward for reinforcement learning system?

Can $\Phi$ measure (computed rigorously or approximately) of Integrated Information Theory serve as reward for self-evolving/learning reinforcement learning system and hence we let this system to ...
3
votes
0answers
217 views

Should noise (such as OU) be decreased over time in actor / critic algorithms?

In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence. But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
3
votes
0answers
31 views

Code examples of controlling multiple units with RL

Anyone knows a resources (papers, articles and especially repositories) regarding controlling multiple units with RL. The controlled units should not be fixed, for example in Real Time Strategy the ...
3
votes
0answers
52 views

Deep Q-Learning agent poor performing actions. Need help optimizing

I'm trying to make deep q-learning agent from https://keon.io/deep-q-learning My environment looks like this: https://imgur.com/a/OnbiCtV As you can see my agent is a circle and there is one gray ...
3
votes
0answers
83 views

Feature Selection using Monte Carlo Tree Search

I'm trying to tackle the problem of feature selection as an RL problem, inspired by the paper Feature Selection as a One-Player Game. I know Monte-Carlo tree search (MCTS) is hardly RL. So, I used ...
3
votes
0answers
192 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
3
votes
0answers
42 views

Does everyone still use discount rates?

In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the ...
3
votes
0answers
57 views

Negative counterfactual regret

I am reading the paper Regret Minimization in Games with Incomplete Information on CFR algorithm. On page 4, the paper defines $R^{T,+}_{i,\text{imm}}=\max\{R^{T}_{i,\text{imm}}, 0\}$ after equation (...
3
votes
1answer
119 views

Can A3C update the policy / critic on a local machine without needing to copy?

To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/...
3
votes
0answers
49 views

Choosing more than one action in a parameterized policy

I would like to implement a variant of policy iteration that can choose one or more actions in each state. An example would be to heal and move in the game of Doom. Parameterizing the power set of ...
3
votes
0answers
106 views

Can I use deterministic policy gradient methods for stochastic policy learning?

Can I treat a stochastic policy (over a finite action space of size $n$) as a deterministic policy (in the set of probability distribution in $\mathbb{R}^n$)? It seems to me that nothing is broken ...
3
votes
0answers
117 views

Why are all the actions converging to the same index?

I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
3
votes
0answers
347 views

Solving equations using reinforcement learning

I was lately curious about a reinforcement learning approach that would solve maths equations. For example, if I have the following equation: $$ f(g(h(w))) = 0 , with \ w = \begin{matrix} a_{11} &...
3
votes
0answers
219 views

Dyna-Q algorithm, having trouble when adding the simulated experiences

I'm trying to create a simple Dyna-Q agent to solve small mazes, in python. For the Q function, Q(s, a), I'm just using a matrix, where each row is for a state value, and each column is for one of the ...
3
votes
0answers
175 views

Is iLQG a good algorithm for model-based planning with simple environments?

In their work Continuous Deep Q-Learning with Model-based Acceleration, the author demonstrate great results of applying Imagination Rollouts for model-based acceleration of learning. They test their ...
3
votes
0answers
399 views

RL to generate sentences

I want to develop a system to generate grammatically correct sentences. The input would be some words. The output would be a grammatically correct human-like sentence. Eg: Input: capital, Paris, ...
2
votes
0answers
65 views

REINFORCE differentiation on sum or single value?

I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : https://towardsdatascience.com/policy-gradient-methods-104c783251e0 that the ...
2
votes
0answers
34 views

How do neural networks deal with inputs of different sizes that are padded in order to have them of the same size?

I am trying to create an environment for RL where the size of my input (observation space) is not fixed. As a way around it, I thought about padding the size to a maximum value and then assigning &...
2
votes
0answers
42 views

How to prove Lemma 1.6 in the book "Reinforcement Learning: Theory and Algorithms"

I am trying to prove the following lemma from Reinforcement Learning: Theory and Algorithms on page 8. Lemma 1.6. We have that: $$ \left[(1-\gamma)\left(I-\gamma P^{\pi}\right)^{-1}\right]_{(s, a),\...
2
votes
0answers
25 views

Doubt in Sutton & Barto's off-policy Monte Carlo control algorithm

The algorithm is described as below: My understanding: In the third last step, we act greedily w.r.t $Q$. Since we use importance sampling, this $Q \approx Q_\pi$. However, in the next step, whenever ...
2
votes
0answers
35 views

Can we combine policy evaluation and value iteration steps for solving model-based MDP?

In Sutton & Barto (2nd edition), at the very end on page 83, the following is mentioned: In general, the entire class of truncated policy iteration algorithms can be thought of as sequences of ...
2
votes
0answers
33 views

Is there a way to beat AlphaGo Zero with different method?

As I read the research from https://deepmind.com/research It seem AlphagoZero use zero knowledge and use Reinforcement learning to improve the ai skill of playing. Is there a way to beat AlphagoZero? ...
2
votes
1answer
39 views

How do I represent sample efficiency of RL rewards in mathematical notation?

I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
2
votes
0answers
43 views

How does the paper implement NEAT without a global set tracking Innovations?

I have been reading this paper on NEAT and trying to implement the algorithm in C#. For the most part, I understand everything in the paper however, there are 2 things I don't understand that confuse ...
2
votes
1answer
48 views

How does sharing parameters between the policy and value functions help in PPO?

The PPO objective may include a value function error term when parameters are shared between the policy and value functions. How does this help, and when to use a neural network architecture that ...
2
votes
0answers
114 views

Update Rule with Deep Q-Learning (DQN) for 2-player games

I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great ...
2
votes
0answers
44 views

Why isn't RL considered a continual learning strategy itself?

I have read about methods that apply continual learning strategies to reinforcement learning. Since reinforcement learning also learns step by step (i.e., task by task, in a sense) during the training ...

1
2
3 4 5
13