14
How does this method differ from the experience replay, as they both use past information in the training? What's the typical application of both techniques?
Using a recurrent neural network is one way for an agent to build a model of hidden or unobserved state in order to improve its predictions when direct observations do not give enough information, but ...
8
Here's an intuitive description answer:
Function approximation can be done with any parameterizable function. Consider the problem of a $Q(s,a)$ space where $s$ is the positive reals, $a$ is $0$ or $1$, and the true Q-function is $Q(s, 0) = s^2$, and $Q(s, 1)= 2s^2$, for all states. If your function approximator is $Q(s, a) = m*s + n*a + b$, there exists no ...
6
DQN and AlphaZero do not share much in terms of implementation.
However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, value, policy, then these are interchangeable between DQN and AlphaZero. When it comes to implementation, and what each part of the system is doing, then ...
5
I have done some research and would like to share.
Generally to eliminate the use of target network one needs to show that training would be stable under off-policy semi-gradient.
There are two approaches that might work:
Experience reweighting
Constrained optimization
Experience reweighting
Probably the simplest idea is to use importance sampling ...
5
We can start with equation (30):
$$
\bar{A}(s) = P(a \neq \tilde{a}) \mathbb{E}_{(a,\tilde{a})\sim(\pi,\tilde{\pi}|a\neq\tilde{a})} [A_\pi(s, \tilde{a}) - A_\pi(s, a)]
$$
Taking the absolute value of both sides, the equality remains true. We can pull the probability term out of the absolute value since it is guaranteed to be nonnegative.
$$
|\bar{A}(s)| = ...
5
I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:
In the PPO paper, after describing the standard policy gradient objective $L^{PG}$, they mention the following:
While it is appealing to perform multiple steps of optimization on this loss $L^{PG}$ using the same trajectory, doing so ...
4
No. DQN and other deep RL methods work well with fully connected layers. Here's an implementation of DQN which doesn't use CNNs: github.com/keon/deep-q-learning/blob/master/dqn.py
DeepMind mostly use CNN because they use image as input state, and that because they tried to evaluate performance of their methods vs humans performance. Humane performance is ...
4
In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.
The larger the experience replay, the less likely you will sample correlated elements, hence the more stable the training of the NN will be. However, a large experience replay ...
4
A neural network with at least one hidden layer is often considered "deep", in the sense that it can approximate any "reasonable" function, given "enough" (but finite number of) units (or neurons) in the layers. See the universal approximation theorem.
However, have a look at this question https://stats.stackexchange.com/q/229619/82135, where a few people ...
4
As far as I'm aware, it is still somewhat of an open problem to get a really clear, formal understanding of exactly why / when we get a lack of convergence -- or, worse, sometimes a danger of divergence. It is typically attributed to the "deadly triad" (see 11.3 of the second edition of Sutton and Barto's book), the combination of:
Function approximation, ...
4
In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do not usually do this, but it does happen sometimes.
I guess that, from a theoretical point of view, it was very easy of the authors to make their algorithm more ...
4
First, the derivative is usually taken with respect to a variable (input) of the function. Hence the notation $\frac{df}{dx}$ for some function $f(x)$.
If you look at your equation more carefully
$$\nabla log P(\tau^{i};\theta) = \Sigma_{t=0}\nabla_{\theta}log\pi(a_{t}|s_t, \theta).$$
You will see that the gradient is taken with respect to $\theta$, which ...
4
The way you have described tends to be the common approach. There are of course other ways that you could do this e.g. using an exponential decay, or to only decay after a 'successful' episode, albeit in the latter case I imagine you would want to start with a smaller $\epsilon$ value and then decay by a larger amount.
4
If you're interested in the theory behind Double Q-learning (not deep!), the reference paper would be Double Q-learning by Hado van Hasselt (2010).
As for Double deep Q-learning (also called DDQN, short for Double Deep Q-networks), the reference paper would be Deep Reinforcement Learning with Double Q-learning by Van Hasselt et al. (2016), as pointed out ...
3
For tabular Q-learning, the q-values for state s and action a are updated according to
$$
Q(s, a) \gets Q(s, a) + \alpha [(r + max_{a'} Q(s', a')) - Q(s,a)]
$$
where $\alpha$ is the learning rate and $(r + max_{a'} Q(s', a')) - Q(s,a)$ is the difference between the current estimate of the q-value, $Q(s,a)$, and the target, $r + max_{a'} Q(s', a')$.
The ...
3
I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the corresponding Q value. However, to obtain $\max_aQ(s', a)$ (which is a term of the update rule of the Q-learning algorithm) you would need a "forward pass" of this network ...
3
The differences you have observed between the two different versions of the TRPO paper are due to different formalizations of the problem and the objective.
In the first version of the paper you linked, they start out in Section 2 by defining Markov Decision Processes (MDPs) as tuples that, among other things, have a cost function $c : \mathcal{S} \...
3
The purpose of Reinforcement Learning is to maximize some notion of cumulative reward, leading me to the point (1) :
as far as I understand, there is no timesteps in your problem and the "reward" is immediate.
Thus, I don't think reinforcement is suitable here.
On an other hand, in supervised learning, linear regression is the task of approximating a ...
3
For the programming part I suggest this YouTube channel by Phil Tabor (he also has a website: neuralnet.ai. I found his videos really useful while I was attending reinforcement learning classes at the uni. He covers basic algorithms like value iteration and policy iteration and also more advanced like deep q learning, covering all main python libraries (...
3
There are several common deep reinforcement algorithms and models apart from deep Q networks (or deep Q learning). I will list them below (along with a link to the paper that introduces them or a resource that describes them).
Proximal policy optimization (PPO)
Trust region policy optimization (TRPO)
Deep deterministic policy gradient (DDPG)
Asynchronous ...
3
Dueling-DQN has different network architecture comparing to vanilla DQN, so I don't think your version will work as well as the Dueling architecture.
From Wang et al., 2016, Dueling Network Architectures for Deep Reinforcement Learning
On the other hand, since we only have the target Q-value, separating the Q-value into state value and advantage result ...
3
The programmer already guides the RL algorithm (or agent) by specifying the reward function. However, the reward function alone may not be sufficient to learn efficiently and fast, as you correctly noticed.
To attempt to solve this inefficiency problem, one solution is to combine reinforcement learning with supervised learning. For example, the paper Deep Q-...
3
Whats does the target Q-values represent?
In a DQN, which uses off-policy learning, they represent a refined estimate for the expected future reward from taking an action $a$ in state $s$, and from that point on following a target policy. The target policy in Q learning is based on always taking the maximising action in each state, according to current ...
3
The deep Q-learning (DQL) algorithm is really similar to the tabular Q-learning algorithm. I think that both algorithms are actually quite simple, at least, if you look at their pseudocode, which isn't longer than 10-20 lines.
Here's a screenshot of the pseudocode of DQL (from the original paper) that highlights the Q target.
Here's the screenshot of Q-...
3
The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does exist and is correct.
It is called soft update and it has been used in the Deep Deterministic Policy Gradient (DDPG) paper, which uses the concept of a target ...
2
In reinforcement learning (RL), an agent interacts with an environment in time steps. At each time step $t$, the agent and the environment are in some state $s_t$. From that state $s_t$, the agent chooses and executes an action $a_t$ and the environment emits a reward $r_t$ (which values the just taken action $a_t$). Finally, the agent and the environment ...
2
Andrej Karpathy's blog has a tutorial on getting a neural network to learn pong with reinforcement learning. His commentary on the current state of the field is interesting.
He also provides a whole bunch of links (David Silver's course catches my eye). Here is a working link to the lecture videos.
Here are demos of DeepMinds game playing.
Get links to the ...
2
After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were consecutives.
After redoing my experience selection method, my DQN is actually working and has reached about +200 points after 400000 experiences (about 500 episodes; ...
2
There are many techniques for training an RL agent without explicitly interacting with an environment, some of which are cited in the paper you linked. Heck, even using experience replay like in the foundational DQN paper is a way of doing this. However, while many models utilize some sort of pre-training for the sake of safety or speed, there are a couple ...
Only top voted, non community-wiki answers of a minimum length are eligible
