New answers tagged reinforcement-learning
0
Evolutionary algorithms (EAs) are a family of algorithms inspired by the biological evolution that can be used to solve (constrained or not) optimization problems where the function that needs to be optimized does not necessarily need to be differentiable (or satisfy any strong constraint). In EAs, you typically only need to define
an encoding of the ...
0
$\text{rank}(f((r,e)|u))$ in $A_t(u)$ means to compute the value of scoring function $f$ for all $(r,e)\in A_t$ which are condition by $u$, then sort them in a descending order. The rank of the $f((r,e)|u)$ in this order is equal to $\text{rank}(f((r,e)|u))$. Hence $\text{rank}(f((r,e)|u)) \leqslant \alpha$ means to select the $\alpha$ top most scored pairs.
0
Firstly, since you are a beginner, I strongly recommend you start reading Sutton's book. It is a really great book.
Then, some tutorials:
udemy rl
udemy deep rl
rl-with-tensorflow
learndatasci
stackabuse
1
There are a couple of things to break down here.
The first thing is to correct this:
For example, the reward for the game tic-tac-toe is decided at the end of the episode, when the player wins, loses, or draws the match. The reward is not available at each step $t$.
In a Markov Decision Process (MDP), there is always an immediate reward for each time $t$...
2
Yes, it is the state of the memory; this would mainly involve variables, since the code would be in ROM. Since it is only 128 bytes in size, the screen memory would also not be included in this.
The idea is that all information relevant to the game is captured in these 128 bytes; they represent the state of the game world at any given time. Movements of the ...
1
In the tabular case, then the Q table will only converge if you have walked around the whole of the table. Note that to guarantee convergence we need $\sum\limits_{n=1}^{\infty}\alpha_n(a) = \infty$ and $\sum\limits_{n=1}^\infty \alpha_n^2(a) < \infty$. These conditions imply that in the limit each state-action pair will have been visited an infinite ...
2
I am aware that as the experience set grows the Central Limit Theorem
will come into play and the distribution of experience will more
accurately represent the true environment's state-actions-rewards
distribution
I believe here you mean the Law of Large Numbers which states that for a large enough sample ($n \rightarrow \infty$) the sample mean will ...
3
Is this a sign that the algorithm diverged?
It is a common sign of a problem with learning process. That includes divergence due to poor hyper-parameters, even just bad luck. But it can also point to a design/architecture problem.
Other common causes of algorithm failing with a fixed action choice include:
Neural network inputs not scaled before use.
...
2
Is the main difference between these two problems, and hence why one is regression and the other is kernel density estimation, because with the reward we are mainly concerned with the expected reward (hence regression) whereas with the state transitioning, we want to be able to simulate this so we need the estimated density?
Yes.
An expected reward ...
0
There are three problems
Limited capacity Neural Network (explained by John)
Non-stationary Target
Non-stationary distribution
Non-stationary Target
In tabular Q-learning, when we update a Q-value, other Q-values in the table don't get affected by this. But in neural networks, one update to the weights aiming to alter one Q-value ends up affecting other Q-...
5
If you're interested in the theory behind Double Q-learning (not deep!), the reference paper would be Double Q-learning by Hado van Hasselt (2010).
As for Double deep Q-learning (also called DDQN, short for Double Deep Q-networks), the reference paper would be Deep Reinforcement Learning with Double Q-learning by Van Hasselt et al. (2016), as pointed out ...
3
You should first read the introductory paper of Double DQN.
https://arxiv.org/abs/1509.06461
Then, depending on what you would like to do, search for other relevant papers that use this method.
1
Per this paper a look ahead policy is a policy that will make decisions based on some 'horizon'. Here horizon means some time steps into the future, and so a finite horizon is simply a finite amount of time steps into the future. For example, as we are typically concerned with maximising returns in RL, a 10-step look ahead policy would choose an action at ...
4
The way you have described tends to be the common approach. There are of course other ways that you could do this e.g. using an exponential decay, or to only decay after a 'successful' episode, albeit in the latter case I imagine you would want to start with a smaller $\epsilon$ value and then decay by a larger amount.
2
Almost certainly, there is no such paper since that would be a trivial problem. The pole lying flat is the definition of failure, hence game over. If you started in that position, you would be permanently in the game-over state and you would never learn anything.
The reason is that if the pole is lying flat, then, if you apply a force on the cart (in the ...
6
The weights do sum to one. Note that in the second line where we have
$$\frac{\epsilon}{|\mathcal{A}(s)|} \sum_a q_{\pi}(s,a) + (1-\epsilon)\max_aq_{\pi}(s,a) \; ,$$
the sum is over the whole action space, including the greedy action, so the sum of the weights will be $\frac{\epsilon}{|\mathcal{A}(s)|} \times |\mathcal{A}(s)| + (1-\epsilon) = 1$.
2
It is difficult to prove a negative, but I doubt there will be a paper on that specific problem. It should be relatively easy to adjust the environment or write a new one that does this if you wished though.
A very similar environment that does have a lot more written about it is Acrobot, which does have a OpenAI Gym version. Instead of a cart on a track ...
0
I think I may be in position to answer my own question. The Bellman equation (for the optimal policy) for a MDP with $r(s,a,s')$ rewards would look like this:
$$V(s) = \max_a \left\{ \sum_{s'} p(s'|s,a)(r(s,a,s') + \gamma V(s')) \right\} $$
$$V(s) = \max_a \left\{ \sum_{s'} p(s'|s,a) \cdot r(s,a,s') + \gamma \sum_{s'} p(s'|a,s) \cdot V(s') \right\} $$
Now, ...
2
You have two options, either interpolate or restrict the actions only to values that produce states which are in your state vector.
The simplest interpolation scheme is a linear interpolation, which works as follows (assuming DS contains a set of grid points in increasing order). For a state $s'$ you can locate its closest neighbours from the array DS and ...
0
There is a recent paper: Continuous-Discrete Reinforcement Learning for
Hybrid Control in Robotics published by DeepMind that aims to solve this problem, as stated in the abstract:
Many real-world control problems involve both discrete decision variables – such as the choice of control modes, gear switching or digital outputs – as
well as continuous ...
3
I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results?
That is correct. If you fix the seed for a process which inherently has stochastic behaviour by design (such as initialising neural network params), then what you know about the model is ...
1
Below are some tweaks that helped me accelerate the training of DDPG on a Reacher-like environment:
Reducing the neural network size, compared to the original paper. Instead of:
2 hidden layers with 400 and 300 units respectively
I used 128 units for both hidden layers. I see in your implementation that you used 256, maybe you could try reducing this. ...
2
Aside from the points raised in nbro's answer, I'd like to point out that for a single MDP (a single instance of a "problem"), it may be sensible to study it from perspectives that include no policy at all, or multiple different policies.
For instance, if I have an MDP, I may be interested in studying it by looking at various inherent properties of the ...
7
The MDP defines the environment (which corresponds to the task that you need to solve), so it defines e.g. the states of the environment, the actions that you can take in those states, the probabilities of transitioning from one state to the other and the probabilities of getting a reward when you take a certain action in a certain state.
The policy ...
0
In the context of reinforcement learning, the idea of modeling your goal-oriented problem as a hierarchy of multiple sub-problems is called hierarchical reinforcement learning, which gives rise to concepts such as semi-Markov decision processes and options (aka macro actions). The article The Promise of Hierarchical Reinforcement Learning presents and ...
2
I think there is an intersection. There are problems that are in reinforcement learning and in learning in multi-agent systems. There are problems in reinforcement learning, but not exactly in multi-agent systems. And there is learning in multi-agent systems that is not through reinforcement learning. For sort you can say: multi-agent reinforcement learning. ...
0
$Q(s,a)$ denotes the $Q-value$ for the state-action pair. It means the expected returns if we start from state $s$, take action $a$, and act according to whatever policy we are currently following.
Suppose we are in state $s_0$, take action $a_0$. To compute the returns, we would need to follow our current policy from whatever state we land up after taking $...
0
So is the baseline expected to be used in the next iteration when our policy has changed?
Yes.
To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?
The precise value of the baseline is not that important. What is ...
2
There are different TD algorithms, e.g. Q-learning and SARSA, whose convergence properties have been studied separately (in many cases).
In some convergence proofs, e.g. in the paper Convergence of Q-learning: A Simple Proof (by Francisco S. Melo), the required conditions for Q-learning to converge (in probability) are the Robbins-Monro conditions
$\sum_{...
2
Recall that our goal is to be able to accurately estimate the true value of each state by computing a sample average over returns starting from that state:
$$v_{q}(s) \doteq \mathbb{E}_{q}\left[G_{t} | S_{t}=s\right] \approx \frac{1}{n} \sum_{i=1}^{n} Return_i $$
where $Return_i$ is the return obtained from the $i^{th}$ trajectory.
The problem is that the $\...
1
Can we make the game faster so that model will be training faster?
It depends on how much processing is required to run the simulation, how efficient that is implemented in whichever library you have loaded, and whether there is anything non-necessary for training that you can disable. Some environments for instance deliberately run "real time" so humans ...
1
In the application of importance sampling to RL, is the expectation of the function $f$ equivalent to the value of the trajectories, which is represented by the trajectories $x$?
I believe what you are asking here is if when using importance sampling in the off-policy RL setting that we set $f(x)$ from the general importance sampling formula to be our ...
1
The first one is the update rule that we use in the $Q$-learning algorithm.
The second one is the "definition" of $Q(s, a)$ values, although I would personally write it as follows, with an expectation around the reward, to also support cases where rewards might be non-deterministic;
$$Q(s, a) \doteq \mathbb{E} \left[ r(s, a) \right] + \gamma \max_a Q(s', a)...
1
First let us note the definition of the advantage function:
$$A(s,a) = Q(s,a) - V(s) \; ,$$
where $Q(s,a)$ is the action-value function and $V(s)$ is the state-value function. In theory you could represent these by two different function approximators, but this would be quite inefficient. However, note that
$$Q(s,a) = \sum_{s',r} \mathbb{P}(s',r|s,a)(r + ...
3
Softmax policy $\pi_\theta(s,a)$ is defined as $\frac{\exp{(\phi(s,a)^T \theta})}{\Sigma \exp{(\phi(s,a) ^T \theta) }}$, where the summation is over the action space.
Taking log, this becomes
$$
\log \pi_\theta(s,a) = log(e^{\phi(s,a) ^T \theta}) - log({\Sigma e^{\phi(s,a) ^T \theta }}) \\
= \phi(s,a) ^T \theta - log({\Sigma e^{\phi(s,a)^T \theta }})
$$
...
0
If your intention is to learn make the agent learn which has the min arbitrary value, then you would need to modify your rewards a bit.
The current reward structure provides the incentive to just move to a stage where it gets a reward.
For example, if it is at state 0, it gets the same reward to go to either state 2 or state 3, since both of them have a ...
3
Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?
Yes, but only if you control all places in the code where stochastic methods are used (typically by seeding the affected RNGs):
Neural network weight initialisation
Action choice for $\epsilon$-greedy or other behaviour policy (does not apply in your case, ...
3
It is common in Bayesian statistics to only know the posterior up to a constant of proportionality. This means that we can't directly sample from the posterior. However, using importance sample we are able to.
Consider our posterior density $\pi$ is only known up to some constant, i.e. $\pi(x) = K \tilde{\pi}(x)$, where $K$ is some constant and we only ...
3
The rationale behind importance sampling is that $q(x)$ is difficult to sample from but easy to evaluate. Or at least you can easily evaluate some $\tilde{q}$ such that:
$$
\tilde{q}(z) = Zq(z)
$$
where $Z$ (scalar) might be unknown. The geometrical example would be here e.g. sampling uniformly from an area under the curve $q(x)$ (in general it's not easy).
...
1
The advantage is basically a function of the actual return received and a baseline. The function of the baseline is to make sure that only the actions that are better than average receive a positive nudge.
One way to estimate the baseline is to have a value function approximator. At every step, you train a NN, using the trajectories collected via the ...
1
In RL, neural networks may intuitively be thought of as using the input features as a representation that "identifies" the input state (or input state + action pair). Think back to the "tabular" RL setting that most people first study when they learn about RL. In tabular RL, you have a table of values (state values $V(s)$, or state-action values $Q(s, a)$), ...
1
The video you linked is not using reinforcement learning (RL). It is using genetic algorithms (GA).
GA is designed around using multiple agents and picking the best performing to move forward to next generation. With this approach, it is common to want to only view the best performing agents, as the learning mechanism uses the same selection process - the ...
1
Model your problem as an MDP
To solve a problem with reinforcement learning, you need to model your problem as a Markov decision process (MDP), so you need to define
the state space,
the action space, and
the reward function
of the MDP.
Understand your problem and the goal
To do define these, you need to understand your problem and define it as a goal-...
2
Here is the commit
I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.
In [2]: histories = [deque(maxlen=5)] * 4
In [3]: histories ...
2
First, some preliminary questions: in this case, what is the optimal policy?
It is the policy that maximises return from any given time step $G_t$. You need to be careful with your definition of return with continuing environments. The simple expected sum of future rewards is likely to be positive or negative infinity.
There are three basic approaches:
...
1
In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge.
It may help if you think through what the actual consequences for making the "wrong" move are in your environment. There are a few self-consistent options:
The move simply cannot be made and count as playing the game as ...
4
Yes, the two update equations are equivalent. As an aside, technically the equation you give is not the Bellman equation, but the update step re-written as an equation - in the Bellman equation instead of $v_{k+1}(s)$ or $v_{k}(s)$ (showing iterations of approximate value functions), you would have $v_{\pi}(s)$ (representing the true value of a state under ...
0
I have not used learning rate schedules but do have experience with adjustable learning rates. The Keras callback ReduceLROnPlateau is useful for adjusting the learning rate.If you use it to monitor the validation loss versus training loss you will avoid the danger of over fitting. Also you can use the ModelCheckpoint callback to save the model with the ...
1
You don't need to manage negative rewards separately, if you implemented the algorithm correctly it will work regardless if the rewards are negative or not. You seem to be using rewards for the loss but you should be using the return which is the sum of the rewards for some state action pair from that point until the end of trajectory.
You also seem to be ...
0
I went into the pytorch code for the spinning up implementation of vanilla policy gradient and from what I could understand, found that they use a learning rate of 1e-3 for training the baseline and do a gradient descent 80 times on the same dataset by default with no termination criteria.
Also it is usually impossible to fit the value function completely ...
Top 50 recent answers are included
