Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
0
votes
0answers
20 views

$\epsilon$-greedy policy in environments where actions are performed in a long term. Does it has influence?

I'm working in an environment where once an action $a \in A$ is performed, it must hold this action selection for a while. To clarify this, assumes a horizon length $h$ and the set of actions: $\{a_{1}...
3
votes
1answer
44 views

Is there a document with a list of conjectures or research problems regarding reinforcement learning (like the Millennium Prize Problems)?

Is there a document with a list of conjectures or research problems regarding reinforcement learning like the Millennium Prize Problems?
1
vote
0answers
41 views

How should I simulate this Markov Decision Process?

I am working on solving a problem on nodes in a graph communicating with each other. They try to estimate a central state using Kalman consensus filter, with the connections described by the graph's ...
2
votes
1answer
36 views

Clarifying representation of Neural Nerwork input for Chess Alpha Zero

In the Alpha Zero paper (https://arxiv.org/pdf/1712.01815.pdf) page 13, the input for the NN is described. In the beggining of the page, the authors state that: "The input to the Neural Network ...
1
vote
0answers
20 views

Is (log-)standard deviation learned in TRPO and PPO or fixed instead?

After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, ...
1
vote
0answers
30 views

Confusion about computing policy gradient with automatic differentiation ( material from Berkeley CS285)

I am taking Berkeley’s CS285 via self-study. On this particular lecture regarding Policy Gradient, I am very confused about the inconsistency between the concept explanation and the demonstration of ...
4
votes
1answer
1k views

Is there a machine learning model that can be trained with labels that only say how “right” or “wrong” it was?

I'm trying to find the name for a model that is used to output a decision (maybe something like right, left, or ...
0
votes
0answers
26 views

Understanding policies in helicopter control in the paper by Andrew Ng et al

I was going through this paper on helicopter flight control using reinforcement learning by Andrew Ng et al. It defines two policy classes to learn two policies, one for hovering the helicopter and ...
2
votes
1answer
61 views

Why do I get the best policy before Q values converge using DQN?

I have implemented DQN algorithm and wonder why during testing, the best performance is achieved by a policy from about 300 episode, when mean Q values converge at about 800 episode? Mean Q-values ...
2
votes
1answer
41 views

What should the input and output of the Q-network be in the case of an ordinal action space?

I recently started looking into implementations of the DQN algorithm (e.g. TensorFlow) in some more detail. All the implementations that I found use a network that gives an output for each possible ...
1
vote
2answers
93 views

What is the target output for updating a Deep Q Network

I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space. The algorithm for table-based Q-Learning updates a single entry of the Q table - ...
1
vote
0answers
27 views

If one of the inputs to a neural network (that represents a policy) is noisy and degrades the performance, would this architecture solve the issue?

I'm using genetic algorithms to train deep reinforcement learning (DRL) agents, similarly to what was done in this paper. DRL policies are therefore represented by deep neural networks, which map ...
1
vote
2answers
126 views

Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?

Assuming we use an MSE cost function of the form $$ \sum_s\mu(s)(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2 = E_{\mu(s)}[(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2])$$ The Stochastic Gradient Descent is used ...
7
votes
2answers
919 views

What are the biggest barriers to get RL in production?

I am studying the state of the art of Reinforcement Learning, and my point is that we see so many applications in the real world using Supervised and Unsupervised learning algorithms in production, ...
1
vote
0answers
20 views

Gradual decrease in performance of a DDPG agent

I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some ...
8
votes
2answers
831 views

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Q-learning uses a table to store all state-action pairs. Q-learning is a model-free RL algorithm, so how could there be the one called Deep Q-learning, as deep means using DNN; or maybe the state-...
3
votes
1answer
56 views

Does DQN generalise to unseen states in the case of discrete state-spaces?

In my understanding, DQN is useful because it utilises a neural network as a q-value function approximator, which, after the training, can generalise to unseen states. I understand how that would work ...
3
votes
0answers
64 views

Where does this variation of the importance sampling weight come from?

I have seeing a variation in importance sampling (IS) in Prioritized Experience Replay (PER) in some implementations regarding the original paper approach stated as (in section 3.4): $$ w_{i}=\left(\...
0
votes
1answer
73 views

How to train a policy model incrementally to solve a problem similar to the vehicle routing problem?

I have a problem similar to the vehicle routing problem (VRP) that I want to solve with reinforcement learning. In this problem, the agent starts from the point $(x_0, y_0)$, then it needs to travel ...
2
votes
0answers
17 views

Can DQN outperform DoubleDQN?

I found a similar post about this issue, but unfortunately I did not find a proper answer. Are there any references where DQN is better than DoubleDQN, that is DoubleDQN does not improve DQN ?
0
votes
0answers
17 views

Which loss function should I use to train DDGP with multiple q values, one for each of the output dimensions?

I'm trying to come up with a loss function for the case, in DDPG, where we have as many outputs from the critic as there are from the actor. So, there will be one Q value for each dimension in the ...
2
votes
1answer
20 views

How does DDPG algorithm know about my action mapping function?

I am using DDPG to solve a RL problem. The action space is given by the Cartesian product $[0,20]^4\times[0,6]^4$. The actor is implemented as a deep neural network ...
3
votes
1answer
66 views

How does AlphaZero's MCTS work when starting from the root node?

From the AlphaGo Zero paper, during MCTS, statistics for each new node are initialized as such: ${N(s_L, a) = 0, W (s_L, a) = 0, Q(s_L, a) = 0, P (s_L, a) = p_a}$. The PUCT algorithm for selecting ...
1
vote
0answers
29 views

Are inputs into AlphaZero the same during the evaluate step in MCTS and during test time?

From the AlphaZero paper: The input to the neural network is an N × N × (M T + L) image stack that represents state using a concatenation of T sets of M planes of size N × N . Each set of planes ...
1
vote
0answers
37 views

DDQN Agent in Othello (Reversi) game struggle to learn

This is my first question on this forum and I would like to welcome everyone. I am trying to implement DDQN Agent playing Othello (Reversi) game. I have tried multiple things but the agent which seems ...
1
vote
1answer
33 views

Is there a training data capacity limit for AlphaZero (Chess)?

In AlphaZero, we collect ($s_t, \pi_t, z_t$) tuples from self-play, where $s_t$ is the board state, $\pi_t$ is the policy, and $z_t$ is the reward from winning/losing the game. In other DeepRL off-...
2
votes
0answers
68 views

Policy gradient: Does it use the Markov property?

To derive the policy gradient, we start by writing the equation for the probability of a certain trajectory (e.g. see spinningup tutorial): $$ \begin{align} P_\theta(\tau) &= P_\theta(s_0, a_0, ...
1
vote
1answer
49 views

How to build a Neural Network to approximate the Q-function?

I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement ...
0
votes
0answers
43 views

In AlphaZero, which features are one-hot encoded and which are single real-valued?

From the AlphaZero paper, the caption of Table S1 (p. 13) Table S1: Input features used by AlphaZero in Go, Chess and Shogi respectively. The first set of features are repeated for each position in a ...
1
vote
0answers
48 views

Stack of Planes as the Action Space Representation for AlphaZero (Chess)

I have a question regarding the action space of the policy network used in AlphaZero. From the paper: We represent the policy π(a|s) by a 8 × 8 × 73 stack of planes encoding a probability ...
2
votes
1answer
44 views

In AlphaZero, do we need to store the data of terminal states?

I have a question about the training data used during the update/back-propagation step of the neural network in AlphaZero. From the paper: The data for each time-step $t$ is stored as ($s_t, \pi_t, ...
1
vote
0answers
48 views

What are the differences between Proximal Policy Optimization versions PPO1 and PPO2?

When Proximal Policy Optimization (PPO) was released, it was accompanied by a paper describing it. Later, the authors at OpenAI introduced a second version of PPO, called PPO2 (whereas the original ...
2
votes
0answers
23 views

CNN for a DQN agent with a 2D matrix state and action as a 2D matrix

I have a custom environment, where the state is a 2D matrix of 11 rows (equals to number of users to satisfy) and 3 columns. Each column can take the value of either 0 or 1, and in each row, there can ...
3
votes
1answer
73 views

How to build a DQN agent with state and action being arrays?

I have a Reinforcement-Learning environment where the state is an array of 0s and 1s with length equals to the number of users the agent must satisfy (11 users). The agent must choose one of 12 ...
3
votes
0answers
37 views

DQN Agent with a 2D matrix as input in Keras

I have a Reinforcement Learning environment where the state is a 2D matrix with 0s and 1s (only one column with the value of 1 in each row). Example: ...
1
vote
0answers
26 views

How to compute the Retrace target for multi-step off-policy Reinforcement Learning?

I am implementing the A3C algorithm and I want to add off-policy training using Retrace but I am having some trouble understanding how to compute the retrace target. Retrace is used in combination ...
1
vote
1answer
40 views

Why are shallow networks so prevalent in RL?

In deep learning, using more layers in a neural network adds the capacity to capture more features. In most RL papers, their experiments use a 2 layer neural network. Learning to Reset, Constrained ...
1
vote
0answers
71 views

Does the order in which the features are concatenated to create the state (or observation) matter?

I'm experimenting with an RL agent that interacts with the following environment. The learning algorithm is double DQN. The neural network represents the function from state to action. It's build with ...
2
votes
0answers
27 views

Is Deep SARSA learning a feasible approach?

I noticed that SARSA has been rarely used in the deep RL setting. Usually, the training for DQN is done off-policy. I think one of the major reasons for this is due to greater sample efficiency in ...
0
votes
0answers
28 views

What are the implications of storing the alternative situation (that could have been experienced) in the replay buffer?

Consider an environment where there are 2 outcomes (e.g. dead and alive) and a discrete set of actions. For example, a game where the agent has 2 guns $A$ and $B$ to shoot a monster (the monster dies ...
1
vote
1answer
41 views

Initialising DQN with weights from imitation learning rather than policy gradient network

In AlphaGo, the authors initialised a policy gradient network with weights trained from imitation learning. I believe this gives it a very good starting policy for the policy gradient network. the ...
3
votes
1answer
58 views

How is MuZero's second binary plane for chess defined?

From the MuZero paper (Appendix E, page 13): In chess, 8 planes are used to encode the action. The first one-hot plane encodes which position the piece was moved from. The next two planes encode ...
3
votes
1answer
52 views

DQN layers when state space and action space are multi dimensional

I have built my own RL environment, where a state is composed of two elements: the agent's position and a matrix of 0s and 1s (1 if a user has requested a service from the agent, 0 otherwise); an ...
1
vote
1answer
49 views

Why do we minimise the loss between the target Q values and 'local' Q values?

I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss ...
0
votes
0answers
19 views

Need suggestion for Reinforcement Learning based visual landing system for quadcopters (UAVs)

I have deep interest in quadcopters. I need ideas for designing of an experiment. I have a programmable quadcopter. I can autonomously land it on a staitonary landing pad with a vision algorithm. ...
1
vote
2answers
163 views

Why not use the target network in DQN as the predictor after training

Target network in DQN is known to make the network more stable, and the loss is like "how good I'm now compared to using the target". What I don't understand is, if the target network is the ...
1
vote
0answers
48 views

How does one stack multiple observations in the input layer of a convolutional neural network?

The paper, Deep Recurrent Q-Learning for Partially Observable MDPs, talks about stacking multiple observations in the input of a convolutional neural network. How does this exactly work? Do the ...
2
votes
1answer
65 views

Why is tree search/planning used in reinforcement learning?

In AlphaGo Zero, MCTS is used along with policy networks. Some sources say MCTS (or planning in general) increases the sample efficiency. Assumed the transition model is known and the computational ...
2
votes
2answers
122 views

How should I define the reward function to solve the Wumpus game with deep Q-learning?

I'm writing a DQN agent for the Wumpus game. Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
1
vote
0answers
44 views

How to design an observation(state) space for a simple `Rock-Paper-Scissor` game?

For weeks I've been working with this toy game of Rock-Paper-Scissor. I want to use a PPO agent learn to beat a computer ...

1
2 3 4 5