Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Hot answers tagged

12 votes
Accepted

Why does DQN require two different networks?

My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one. It has no real impact on computation ...
user avatar
  • 23.3k
9 votes

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN). Tabular Q-learning (TQL) Deep Q-learning (DQL) ...
user avatar
  • 33.2k
8 votes
Accepted

Is Experience Replay like dreaming?

The speaker argued that a dream is a random addition of memories, just as experience replay. The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. ...
user avatar
  • 23.3k
8 votes
Accepted

How to combine backpropagation in neural nets and reinforcement learning?

Gradient descent and back-propagation In deep learning, gradient descent (GD) and back-propagation (BP) are used to update the weights of the neural network. In reinforcement learning, one could map (...
8 votes

Can Q-learning be used for continuous (state or action) spaces?

Q-learning for continuous state spaces Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
user avatar
  • 23.3k
6 votes
Accepted

How should I model all available actions of a chess game in deep Q-learning?

To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can ...
user avatar
6 votes
Accepted

My DQN is stuck and can't see where the problem is

After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were ...
user avatar
  • 153
6 votes
Accepted

What is the difference between DQN and AlphaGo Zero?

DQN and AlphaZero do not share much in terms of implementation. However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
user avatar
  • 23.3k
6 votes

Which kind of prioritized experience replay should I use?

The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they ...
user avatar
6 votes
Accepted

What are some online courses for deep reinforcement learning?

Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer ...
user avatar
  • 33.2k
5 votes
Accepted

Is the DQN only applicable with images as inputs?

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)? No, DQN is not restricted to images or other kinds of inputs with those properties, it can be used ...
user avatar
  • 9,326
5 votes
Accepted

When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?

Gradient descent should be performed using the sum (or average) of the losses in the minibatch. This is in fact also how I read the pseudocode in your question, though I understand it can be confusing....
user avatar
  • 9,326
5 votes
Accepted

What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?

1) Is there any way to set the initial Q-values for the actions? You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network ...
user avatar
5 votes

Why are reinforcement learning methods sample inefficient?

I will try to give a broad answer, if it's not helpful I'll remove it. When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model ...
user avatar
5 votes

What are some online courses for deep reinforcement learning?

For the programming part I suggest this YouTube channel by Phil Tabor (he also has a website: neuralnet.ai. I found his videos really useful while I was attending reinforcement learning classes at the ...
user avatar
5 votes
Accepted

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance. Therefore, methods used in RL to solve ...
user avatar
  • 23.3k
5 votes
Accepted

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as $$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, ...
user avatar
4 votes

What are good learning strategies for Deep Q-Network with opponents?

In a two player zero-sum game (if I win, you lose and vice-versa), then you can have a simple and efficient solution learning from self-play. How should an opponent be implemented to get good and ...
user avatar
  • 23.3k
4 votes

Why does self-playing tic-tac-toe not become perfect?

There are lots of ways that RL agents can fail to learn properly, so you are faced with a little bit of experimentation and maybe bug hunting unfortunately. However, from the description you have ...
user avatar
  • 23.3k
4 votes
Accepted

Can DQN announce it has things in its hand in a card game?

The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple. However, if you want to ...
user avatar
  • 23.3k
4 votes
Accepted

What is a high dimensional state in reinforcement learning?

Usually when people write about having a high-dimensional state space, they are referring to the state space actually used by the algorithm. Suppose my state is a high dimensional vector of $N$ ...
user avatar
  • 9,326
4 votes
Accepted

Why does Deep Q Network outputs multiple Q values?

I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the ...
user avatar
  • 33.2k
4 votes
Accepted

Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do ...
user avatar
  • 9,326
4 votes
Accepted

How do we compute the target value when the agent ends up in the terminal state?

Now, let us assume the agent is in the penultimate state, $S_1$, and chooses the action $A$ that leads him to the completion state, $S_2$, and gets a reward $R$. How do we form the target value $Q_\...
user avatar
  • 23.3k
4 votes
Accepted

Should I grey-scale the coloured frames/channels to build the approximation of the state?

We can't say for sure which approach would work best in the general case. If you have domain knowledge, you can make a better guess. You'll basically want to answer the question: which information is ...
user avatar
  • 9,326
4 votes

Why are reinforcement learning methods sample inefficient?

This is mostly because humans already have information when they start learning the game (priors) that makes them learn it more quickly. We already know to jump on monsters or avoid them or to get ...
user avatar
4 votes

Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters?

Dueling-DQN has different network architecture comparing to vanilla DQN, so I don't think your version will work as well as the Dueling architecture. From Wang et al., 2016, Dueling Network ...
user avatar
  • 226
4 votes

What is the target Q-value in DQNs?

When training a Deep Q network with experienced replay, you accumulate what is known as training experiences $e_t = (s_t, a_t, r_t, s_{t+1})$. You then sample a batch of such experiences and for each ...
user avatar
  • 1,201
4 votes
Accepted

How and when should we update the Q-target in deep Q-learning?

The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does ...
user avatar
4 votes
Accepted

What happens when you select actions using softmax instead of epsilon greedy in DQN?

DQN on the other hand, explores using epsilon greedy exploration. Either selecting the best action or a random action. This is a very common choice, because it is simple to implement and quite robust....
user avatar
  • 23.3k

Only top scored, non community-wiki answers of a minimum length are eligible