Questions tagged [reinforcement-learning]
For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.
648
questions
0
votes
0answers
4 views
DQN unlearns certain OpenAI-Gym environments
I solved the OpenAI-Gym MountainCar-v0 environment using dqn(using low-state-dimensional input). When I used the same code for solving CartPole-v0 environment, the network got trained in the reverse ...
3
votes
0answers
27 views
What is the difference between random and sequential sampling from the reply memory?
I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure.
...
0
votes
0answers
21 views
Why doesn't stability in prediction imply stability in control in off-policy reinforcement learning?
Prediction's goal is to get an estimate of a performance of a policy given a specific state.
Control's goal is to improve the policy wrt. the prediction.
The alternation between the two is the ...
3
votes
1answer
31 views
What are the differences between stability and convergence in reinforcement learning?
The terms are mentioned in the paper: “An Emphatic Approach to the Problem of off-Policy Temporal-Difference Learning.” (Sutton, Mahmood, White; 2016) and more, of course.
In which paper, they ...
-1
votes
1answer
23 views
Does higher Accuracy in Reinforcement Learning indicate better model performance?
If a reinforcement learning algorithm uses a Deep Neural Network to predict the action given a state (a NN for a policy function), an Monte Carlo Tree Search in a model-based learning setup, then ...
0
votes
0answers
13 views
A2C for the game of Hanabi underfits
I am trying to solve the game of Hanabi (paper describing game) with actor-critic algorithm. I took code for the environment from the Deepmind's repository and implemented a2c algorithm myself. From ...
2
votes
1answer
31 views
How do policy gradients compute an infinite probability distribution from a neural network
Do neural networks compute the probability distribution for policy gradient methods. If so, how do they compute an infinite probability distribution? How do you represent a continuous action policy ...
2
votes
1answer
41 views
What happens to the optimal value function if the reward is multiplied by a constant?
What happens to the optimal action-value function, $q_*$ if the reward is multiplied by a constant $c$? Is the optimal action-value function also multiplied by such a constant?
2
votes
1answer
56 views
Monte Carlo learning for Reinforcement learning
When you train a model using Monte Carlo-based learning the state and action taken at each step is recorded, and then at some point an end state is reached and the agent receives some reward - what do ...
2
votes
1answer
40 views
Can an RL algorithm trained in one environment be successful in a different one?
Can an RL algorithm trained in one environment be successful in a different one?
For example, if I train a model to go through one labyrinth, could this model also go through a different but similar ...
0
votes
1answer
56 views
How can the derivative of a neural network be calculated, given no mathematical expression?
Neural networks (NNs) are used as approximators in reinforcement learning (RL). To update the policy in RL, the actor network's gradients w.r.t its weights are needed. Since NN doesn't have a ...
5
votes
1answer
32 views
Reinforcement learning with incremental action space
Let's say we have a problem which can be solved by some RL algorithms (DQN, for example, because we have discrete action space). At first, the action space is fixed (the number of actions is $n_1$), ...
1
vote
1answer
26 views
How is the actor-critic algorithm guaranteed to converge?
From my understanding, the critic evaluates the policy (actor) following dynamic programming (DP) or approximate dynamic programming (ADP) scheme, which should converge to the optimal value function ...
1
vote
0answers
48 views
Is Value Iteration better than Policy Iteration for first few iterations?
In Policy Iteration (PI), the action generated by the policy, whether it's optimal or not w.r.t the current value function $v(s)$. Whereas, in Value Iteration, the action is greedily generated w.r.t ...
1
vote
1answer
27 views
Can policy iteration use only the immediate reward for updates?
Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?
3
votes
1answer
31 views
How important is the choice of the initial state?
Is it crucial to always have the same initial (starting) state for Reinforcement Learning, for example, for Q-learning or DQN?
Or it can vary?
3
votes
1answer
51 views
What is a non-starving policy in reinforcement learning?
In the paper, Eligibility Traces for off-Policy Policy Evaluation (2010), by Doina Precup et al., mentioned the term "non-starving" many times. The specific use of the term was like "non-starving ...
3
votes
1answer
31 views
Reinforcement learning with hints or reference model
In Reinforcement Learning, when I train a model, it comes up with its own set of solutions. For example, if I am training a robot to walk, it will come up with its own walking gait, such as this Deep ...
0
votes
0answers
12 views
Invalid moves in Deep Reinforcement Learning for games [duplicate]
I've been working on a bot for a game involving dice throws and chance. The architecture involved is similar to AlphaZero in the that it has Convolutions and MCTS.
According to the current state ...
1
vote
0answers
20 views
PPO: action std or entropy for exploration?
When trying to implement my own PPO (Proximal Policy Optimizer), I came accross two different implementations :
Exploration with action std :
Collect trajectories on ...
3
votes
1answer
68 views
How is the gradient of the loss function in DQN derived?
In the original DQN paper, page 1, the loss function of the DQN is
$$
L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim U(D)} [(r+\gamma \max_{a'} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}))^2]
$$
...
2
votes
1answer
45 views
Can someone please help me validate my MDP?
Problem Statement :
I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if ...
0
votes
0answers
28 views
Finding optimal Value function and Policy for an MDP
I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...
4
votes
1answer
66 views
Benchmarks for reinforcement learning in discrete MDPs
To compare the performance of various algorithms for perfect information games, reasonable benchmarks include reversi and m,n,k-games (generalized tic-tac-toe). For imperfect information games, ...
2
votes
1answer
41 views
Can I have different rewards for a single action based on which state it transitions to?
I am working on an MDP where there are four states and ten actions. I am supposed to derive the optimal policy to reach the desired state. At any state, a particular action can take you to any of the ...
2
votes
0answers
36 views
Feasibility of using machine learning to obtain self-consistent solutions
I am a physicist and I don't have much background on machine learning or deep learning except taking a couple of courses on statistics. In physics, we often simulate a model by means of two-way ...
2
votes
1answer
36 views
Is a state that includes only the past n-step price records partially observable?
I'm currently working on a project to make an DQN agent that decides whether to charge or discharge an electric vehicle according to hourly changing price to sell or buy. The price pattern also varies ...
3
votes
1answer
44 views
Why are model-based methods more sample efficient than model-free methods?
Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two ...
2
votes
1answer
29 views
Can supervised learning be recast as reinforcement learning problem?
Let's assume that there is a sequence of pairs $(x_i, y_i), (x_{i+1}, y_{i+1}), \dots$ of observations and corresponding labels. Let's also assume that the $x$ is considered as independent variable ...
1
vote
1answer
42 views
When does AlphaZero play suboptimal moves?
If AlphaZero was always playing the best moves it would just generate the same training game over and over again. So where does the randomness come from? When does it decide not to play the most ...
4
votes
2answers
131 views
What does the symbol $\mathbb E$ mean in these equations?
I came across some papers that use $\mathbb E$ in equations, in particular, this paper: https://arxiv.org/pdf/1511.06581.pdf. Here is some equations from the paper that uses it:
$Q^\pi \left(s,a \...
2
votes
0answers
24 views
Should importance sample weighting be compensated for by dynamically increasing learning rate?
I'm using Prioritized Experience Replay (PER) with a DDQN. To compensate for overfitting relatively high-value samples due to the non-uniform selection, I'm training with sample weights provided along ...
3
votes
1answer
35 views
Can exogenous variables be state features in reinforcement learning?
I have a question about state representation of Q-learning or DQN algorithm.
I'm still a beginner of RL, so I'm not sure that is it suitable to take exogenous variables as state features.
For example,...
0
votes
0answers
18 views
Loss reduction, but constant performance with CNN
I made a CNN with a reasonable loss curve, but the performance of the model does not improve. I have tried making the model larger, I am using three convolutional layers with batch norms.
Thanks for ...
1
vote
0answers
42 views
Torch CNN not training
I am completely new to CNN's, and I do not quite know how to design or use them efficiently. That being said, I am attempting to build a CNN that learns to play Pac-man with reinforcement learning. I ...
2
votes
0answers
22 views
Developmental systems that try to explain or understand the reward value in the reinforcement learning?
Are there methods (possibly logical or (how they are called in the literature) relational) that allows for the developmental systems to understand or explain the value of the received reward during ...
0
votes
0answers
19 views
Suggestion on How to Push SARS to Memory Buffer of Losing RL Agent in Adversarial Learning
I am trying to implement my first RL program where there are multiple agents, rather than just one. The environment I am using is the connect four game, which is turn-based.
In DQ-Learning, an agent ...
0
votes
0answers
11 views
DDPG: how to implement continous action space bounded in the interval [-2, 2]
I am newbie in reinforcement learning and trying to understand how to implement continuous actions bounded by [-2, 2]. My research shows that doing nothing is a possible solution (i.e. action of 4.5 ...
2
votes
0answers
41 views
How to show Monte Carlo methods converge to an estimate which minimizes mean squared error?
In chapter six of Sutton and Barto (p.128), they claim Monte Carlo methods converge to an estimate minimizing the mean squared error. How can this be shown formally?
Bump
3
votes
1answer
91 views
How to show temporal difference methods converge to MLE?
In chapter 6 of Sutton and Barto (p. 128), they claim temporal difference converges to the maximum likelihood estimate (MLE). How can this be shown formally?
1
vote
3answers
82 views
Reinforcement learning: How to deal with illegal actions?
I'm a beginner of RL and currently trying to make DQN agent that can act optimally in a simple situation.
In the situation agent should decide at what rate to charge or discharge the electrical ...
0
votes
1answer
35 views
LSTM in reinforcement learning
Please tell me that is the LSTM network for the problem of reinforcement learning, as I explain to her what she will get the reward of a prediction, because the output will contain only actions?
Well,...
4
votes
1answer
56 views
Is tabular Q-learning considered interpretable?
I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent ...
0
votes
2answers
33 views
How should I interpret the weights file of the Leela Zero neural network?
I am trying to understand the NN architecture given at https://github.com/leela-zero/leela-zero/blob/next/training/caffe/zero.prototxt.
So, I downloaded the NN weights from http://zero.sjeng.org/. ...
2
votes
1answer
561 views
Should RL rewards diminish over time?
Should a reward be cumulative or diminish over time?
For example, say an agent performed a good action at time $t$ and received a positive reward $R$. If reward is cumulative, $R$ is carried on ...
1
vote
1answer
28 views
Do I need to store the policy for RL?
I am creating a zero-sum game with RL and wondered if I need to store the policy, or if there are other RL methods that produce similar results (consistently beating the human player) without the need ...
2
votes
1answer
44 views
How to evaluate an RL algorithm when used in a game?
Very new to the world of ML and looking for some pointers. I'm planning to create a web-based RL board game and wondered how I would evaluate the performance of the RL part of the application? How ...
0
votes
1answer
27 views
What is the correct name for state explosion from sensor discretization?
The position of a robot on a map contains of an x/y value, for example $position(x=100.23,y=400.78)$. The internal representation of the variable is a 32bit float which is equal to 4 byte in the RAM ...
2
votes
0answers
52 views
Policy gradient methods for continuous action space
I have a problem I would like to tackle with RL but I am not sure if it is even doable. My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
1
vote
0answers
38 views
Has anyone been able to solve OpenAI's hardcore bipedal walker with their implementation of DDPG?
As the question suggests, I'm trying to see if I can solve OpenAI's hardcore version of their gym's bipedal walker using OpenAI's DDPG algorithm.
Below is a performance graph from my latest attempt, ...
