Questions tagged [reinforcement-learning]
For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.
1,241
questions
0
votes
0answers
2 views
ε-greedy policy: what does the term | A(s) | mean?
I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the $|A(s)|$ mean. I guess $A$ is the action set but I'm not sure about that ...
0
votes
0answers
6 views
Training a neural network for games with dynamic starting locations
I've been training a snake for the snake game with Dueling Double DQN as well as the Double DQN. I've noticed that I can see progress being made when the snake starts off from the same location every ...
1
vote
0answers
20 views
Why does the Dyna-Q+ algorithm converge?
I can't understand why Dyna-Q+ algorithm converges, it only gives extra reward for visiting sates that were visited long before, but how does the agent know that the particular state is visited long ...
1
vote
1answer
15 views
How to optimize neural network parameters with REINFORCE
I've seen a few mentions in papers that neural network parameters can be found using REINFORCE algorithm. It was mentioned in the context of nondifferentiable operations involving e.g. step function ...
3
votes
1answer
78 views
What is meant by “generate the data” in describing the difference between on-policy and off-policy?
From the book:
Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. "
following is stated:
"...
2
votes
0answers
28 views
How can I formulate a prediction problem (given labeled data) as an RL problem and solve it with Q-learning?
One of my friends sent me a problem he was working on lately, and I couldn't help but I wonder how could it be solved using Q-learning. The statement is as follows:
Given the following datasets, the ...
2
votes
1answer
37 views
How can I classify policy gradient methods in RL?
In the book of Barto and Sutton, there are 3 methods presented that solve an RL problem: DP, Monte Carlo, and TD. But which category does policy gradient methods (or actor-only methods) classify in? ...
1
vote
1answer
34 views
Why does adding another network help in double DQN?
What is the idea behind double DQN?
The Bellman equation used to calculate the Q values to update the online network follows the equation:
...
2
votes
1answer
31 views
Should forecasting with neural networks only be treated as a supervised learning (regression) problem?
I have recently made a work about the application of neural networks to time series forecasting, and I treated this as a supervised learning (regression) problem. I have come across the suggestion of ...
0
votes
1answer
28 views
How to “forward” updated NN model to a transferred model?
I've trained a robot to walk in a straight line for as long as it can (using TD3), and now I'm using that pre-trained model for two new models with separate purposes: 1. Walk to a specific point and ...
1
vote
1answer
38 views
Can rewards be decomposed into components?
I'm training a robot to walk to a specific $(x, y)$ point using TD3, and, for simplicity, I have something like ...
1
vote
1answer
23 views
If the transition model is available, why would we use sample-based algorithms?
Sample-based algorithms, like Monte Carlo Algorithms and TD-Learning, are often presented as useful since they do not require a transition model.
Assuming I do have access to a transition model, are ...
1
vote
0answers
17 views
Solving multi-armed bandit problems with continuous action space
My problem has a single state and an infinite amount of actions on a certain interval (0,1). After quite some time of googling I found a few paper about an algorithm called zooming algorithm which can ...
-1
votes
0answers
24 views
GAN image inpainting
I am a beginner in GAN. I had this question that for an image inpainting problem using GAN can we set random seed as input or we have to set masked images as input? and how do we basically get the ...
0
votes
1answer
27 views
How should we interpret all the different metrics in reinforcement learning?
I'm trying to train some deep RL agents using policy gradient methods like AC and PPO. While training, I have a ton of different metrics being monitored.
I understand that the ultimate goal is to ...
2
votes
1answer
56 views
How is the reward in reinforcement learning different from the label in supervised learning problems?
How is the notion of immediate reward used in the reinforcement learning different from the notion of a label we find in the supervised learning problems?
1
vote
0answers
13 views
Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?
I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below.
Here, $W$ (importance sampling ratio) is updated at the end of the <...
0
votes
1answer
35 views
How does training for DQN work if messing up in the environment in costly?
Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works.
Is it that ...
2
votes
1answer
41 views
Why do some DQN implementations not require random exploration but instead emulate all actions?
I've found online some DQN algorithms that (in a problem with a continuous state space and few actions, let's say 2 or 3), at each time step, compute and store (in the memory used for updating) all ...
0
votes
0answers
31 views
What does the notation “for t=T to 1,−1 do” in terms of time steps, in deep recurrent q network?
In looking at an algorithm in the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning.
Here is the full algorithm:
What does the notation ...
2
votes
1answer
65 views
Into which subcategories can reinforcement learning be divided?
In the course of a scientific work, I will discuss the different types of reinforcement learning. However, I have difficulties to find these different types.
So, into which subcategories can ...
1
vote
0answers
22 views
What is the role of embeddings in a deep recurrent Q network?
When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning
each agent consists of a recurrent ...
1
vote
0answers
27 views
Do we need multiple parallel environments to train in batches an on-policy algorithm?
When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
0
votes
0answers
16 views
DQN Tic-Tac-Toe does not quite become optimal
I am trying to implement a DQN agent for playing standard 3x3 Tic-Tac-Toe (it is a double DQN with experience replay, and using a target network). I got the hyperparameters to the point
where the ...
0
votes
1answer
35 views
What reinforcement learning algorithm should I use in continuous states?
I want to use reinforcement learning in an environment I made. The exact environment doesn't really matter, but it comes down to this: The amount of different states in the environment is infinite e.g....
0
votes
0answers
21 views
Why is my DDPG agent (implemented in TensorFlow) not learning?
I am trying to implement a Reinforcement Learning algorithm called DDPG in TensorFlow 2.x on a custom gym environment. I am new to TF. So, I started with the DDPG TF 1.x implementation from pemami4911....
1
vote
0answers
12 views
Tic-tac-toe: How would standard SARSA and Q-learning yield different results in the agent's behaviour?
I know this is deceptively simple. Tic tac toe is a well studied game for RL.
Assume your agent is playing aggainst a strong opponent.
I know you deal in after states. I know that in Q learning the ...
2
votes
1answer
33 views
Why does shifting all the rewards have a different impact on the performance of the agent?
I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
1
vote
0answers
20 views
How are the coefficients of the Region of Interest being selected?
I was reading the following paper: Rl-Ncs: Reinforcement Learning Based Data-Driven Approach For Nonuniform Compressed Sensing, and my question is: how do they decide whether a signal is characterized ...
0
votes
0answers
17 views
Generalizability of Generative Adversarial Imitation Learning (GAIL) method
I have something would like to clarify regarding Generative Adversarial Imitation Learning (GAIL). Is the original GAIL applicable if the experts trajectories (sample data) are for the same task but ...
2
votes
1answer
27 views
Should the agent play the game until the end or until the winner is found?
I'm using the DQN algorithm to train my agent to play a turn-based game. The winner of the game can be known before the game is over. Once the winning condition is satisfied, it cannot be reverted. ...
1
vote
2answers
40 views
How do I calculate the return given the discount factor and a sequence of rewards?
I know that $G_t = R_{t+1} + G_{t+1}$.
Suppose $\gamma = 0.9$ and the reward sequence is $R_1 = 2$ followed by an infinite sequence of $7$s. What is the value of $G_0$?
As it's infinite, how can we ...
1
vote
0answers
32 views
Atari Games: Pretrained CNN to accelerate training?
DQN for Atari takes considerable training time. For example, the 2015 paper in Nature notes that algorithms are trained for 50 million frames or equivalently around 38 days of game experience in total....
0
votes
0answers
23 views
Why doesn't my double deep Q network trained with the same training set give consistent performance?
I've written a Double DQN which can do either 1-step or multi-step learning. It also has a prioritised reply buffer. The internal network is an LSTM.
My input data is a series of time series data and ...
2
votes
1answer
57 views
Why is 100% exploration bad during the learning stage in reinforcement learning?
Why can't we during the first 1000 episodes allow our agent to perform only exploration?
This will give a better chance of covering the entire space state. Then, after the number of episodes, we can ...
1
vote
0answers
19 views
Off-Policy Estimation - Importance Sampling with Negative Rewards
Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
1
vote
0answers
17 views
What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?
The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the ...
0
votes
1answer
20 views
Should I build an environment from scratch myself or it is not always needed?
I am inspired by the paper Neural Architecture Search with Reinforcement Learning to use reinforcement learning for optimizing a child network (learner). My meta-learner (controller or parent network) ...
0
votes
1answer
20 views
How is trajectory sampling different than normal (importance) sampling in reinforcement learning?
I am using Sutton and Barto's book for Reinforcement Learning.
In Chapter 8, I am having difficulty in understanding the Trajectory Sampling.
I have read the particular section on trajectory sampling (...
2
votes
1answer
48 views
Implementing SARSA for a 2-stage Markov Decision Process
I am a bit confused as to how exactly I should be implementing SARSA (or Q-learning too) on what is a simple 2-stage Markov Decision Task. The structure of the task is as follows:
Basically, there ...
3
votes
1answer
61 views
Has reinforcement learning been used to prove mathematical theorems?
Coq exists, and there are other similar projects out there. Further, Reinforcement Learning has made splashes in the domain of playing games (a la Deepmind & OpenAI and other less well-known ...
2
votes
1answer
29 views
Should illegal moves be excluded from loss calculation in DQN algorithm?
I'm implementing DQN algorithm to train my agent to play a turn-based game. The action space for the game is small, but not all moves are available at all the states. Therefore, when deciding on which ...
0
votes
0answers
8 views
How to use Tensorflow in Project Malmo [migrated]
I've taken a tensorflow for beginner course and I'm wondering if Project Malmo can be use with Tensorflow, just a simple demonstration will do. Specifically, I'm trying to modify tutorial_6 of python-...
1
vote
0answers
17 views
Why do bootstrapping methods produce nonstationary targets more than non-bootstrapping methods?
The following quote is taken from the beginning of the chapter on "Approximate Solution Methods" (p. 198) in "Reinforcement Learning" by Sutton & Barto (2018):
reinforcement ...
2
votes
0answers
17 views
Why is it the case that off - policy evaluation using importance sampling suffers from high variance?
The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}...
2
votes
1answer
55 views
q learning appears to converge but does not always win against random tic tac toe player
q learning is defined as:
Here is my implementation of q learning of the tic tac toe problem:
...
2
votes
1answer
31 views
Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?
Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference?
Here's the update formula.
3
votes
2answers
75 views
Why is it not advisable to have a 100 percent exploration rate? [duplicate]
During the learning phase, why don't we have a 100% exploration rate, to allow our agent to fully explore our environment and update the Q values, then during testing we bring in exploitation? Does ...
0
votes
1answer
33 views
Why do we update the weights of the target network in deep Q learning?
I know we keep the target network constant during training to improve stability, but why exactly are we updating the weights of our target network? In particular, if we've already reached convergence, ...
0
votes
0answers
33 views
DQN flappybird does not learn
I'm trying to implement DQN to play flappy bird. The action space contains two actions: flap and do nothing. The input for the network is four stacked $80 \times 80$ images.
I've been playing with the ...
