New answers tagged reinforcement-learning
0
According to http://tianlinliu.com/files/notes_exercise_RL.pdf, MDP may not be feasible to multi-target tasks.
In contrast, EA-based methods like NSGA-II, NSGA-III, can solve the multi-target tasks.
And also, tasks that need more than one state to predict the next action are also not suitable to use MDP. For example, when we predict the next action a ...
0
In framing the problem as an episodic reinforcement learning problem, the goal is to find a policy that optimizes $\mathbb{E}[\sum_{t=0}^\tau r(s_t)],$ where $\tau$ is the random time at which the robot leaves the maze. This implicitly assigns a reward of 0 to the out-of-maze state, $s_{terminal}$. If you include this state then the transformation $r\...
0
I've been working with a TD3 implementation for bipedal hardcore. It solved the easy version (v2 and v3) in about 300 epochs (https://github.com/QasimWani/policy-value-methods). I've been training it for hardcore and even after about 1200 episodes, it's no where close to convergence.
Did you end up solving, and if so, what algorithm did you end up going with?...
1
why is it not possible to suggest a policy solely on the basis of state-values; why do we need state-action values?
A policy function takes state as an argument and returns an action $a = \pi(s)$, or it may return a probability distribution over actions $\mathbf{Pr}\{A_t=a|S_t=s \} =\pi(a|s)$.
In order to do this rationally, an agent needs to use the ...
3
The Markov assumption is used when deriving the Bellman equation for state values:
$$v(s) = \sum_a \pi(a|s)\sum_{r,s'} p(r,s'|s,a)(r + \gamma v(s'))$$
One requirement for this equation to hold is that $p(r,s'|s,a)$ is consistent. The current state $s$ is a key argument of that function. There is no adjustment for history of previous states, actions or ...
3
Suppose you learned your action-value function perfectly. Recall that the action-value function measures the expected return after taking a given action in a given state. Now, the goal when solving an MDP is to find a policy that maximizes expected returns. Suppose you're in state $s$. According to your action-value function, let's say actions $a$ maximizes ...
3
In the policy gradient theorem, we don't need to write $r$ as a function of $a$ because the only time we explicitly 'see' $r$ is when we are taking the expectation with respect to the policy. For the first couple lines of the PG theorem we have
\begin{align}
\nabla v_\pi(s) &= \nabla \left[ \sum_a \pi(a|s) q_\pi (s,a) \right] \;, \\ &= \sum_a \left[ \...
3
If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with this reward function is that it's too sparse, meaning the agent has little guidance on how to modify their behavior to become better at attaining said goal, ...
4
Designing reward functions
Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three outcomes: win (good), loss (bad), or draw (neutral). So, you could reward the agent with $+1$ if it wins the game, $-1$ if it loses, and $0$ if it draws (or for any ...
0
Trajectory size can be fixed, but in this case problem would be formulated as something similar to the multi-armed bandit problem where there is a single state and a set of actions to choose from. There is no sequential decision making since samples are not correlated, they are picked at random. So, if you take a batch of 20 examples then you would basically ...
0
According to this paper (PEARL):
These locomotion task families require adaptation across reward
functions (walking direction for Half-CheetahFwd-Back, Ant-Fwd-Back,
Humanoid-Direc-2D, target velocity for Half-Cheetah-Vel, and goal
location for Ant-Goal2D) or across dynamics (random system parameters
for Walker-2D-Params).
It looks like different versions ...
1
However, I’m not sure which policy is saved
The policy from the Monte Carlo tree search is stored, as we can get the policy estimate from the network later by passing the given state through the network, which is used to calculate the cross entropy loss to update the network's policy (summed with Mean squared error loss between value head's prediction and ...
0
My understanding is that it will be the same as p(s' | s, a) for any s, a, s', r combination.
The reward r(s, a, s') is already defined in terms of s, a, s'.
Since p(s', r| s, a) = p(r|s', a, s)* p(s'| a, s). For each case the p(r | s', a, s) is equal to 1 by definition. Thus, the column for p(s', r| a, s) = p(s'| s, a)
For instance,
If,
S= high, A=search ...
0
Since most policies depend solely on actions and states/observations, then if you model the space of this field as the Cartesian Product of your state and action spaces, then the policy learns a surface over this combined space, similar to the way a field is parameterized.
The policy an agent learns could exhibit the same behavior as the field you describe ...
1
I'm using my own implementation of A2C (Advantage Actor Critic) in an industrial application based on Markov Process (present state alone provides sufficient knowledge to make an optimal decision). It's simple and versatile, its performance proven in many different applications. The results so far have been promising.
One of my colleagues had issues with ...
3
You are referring to catastrophic forgetting which could be an issue in any neural net. More specifically for DQN refer to this article.
1
The difference between Vanilla Policy Gradient (VPG) with a baseline as Value function and Advantage Actor Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA:
The value estimates used in updates for VPG are based on full sampled returns, calculated at the end of episodes.
The value estimates used in updates for A2C are ...
1
The policy doesn't change over time. That is, the values will change, otherwise we would not be learning anything, but our rules for action selection don't. I.e. we always take action according to the distribution postulated to our current estimate of the policy $\pi_\theta(a|s)$, we don't suddenly start taking $\max_a \pi_\theta(a|s)$, which would be a true ...
3
You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem and $\mathcal{I}$ to be the set of possible items.
Provided both of these sets are finite, then you should still be able to approach this problem with DQN. ...
0
It is not so much the problem of using Reinforcement Learning to train the neural networks, it is the assumptions made about the data given to standard Neural Networks. They are not capable of handling strongly correlated data which is one of the motivations for introducing Recurrent Neural Networks, as they can handle this correlated data well.
2
The unrolling step is due to the fact you end up with an equation that you can keep expanding indefinitely.
Note that we start with calculating $\nabla v_\pi(s)$ and arrive at
$$\nabla v_\pi(s) = \sum_a\left[ \nabla \pi(a|s) q_\pi(s,a) + \pi(a|s) \sum_{s'}p(s'|s,a) \nabla v_\pi (s') \right]\;,$$
which contains a term for $\nabla v_\pi(s')$. This is a ...
1
This is not quite the loss that is stated in the paper.
For standard policy gradient methods the objective is to maximise $v_{\pi_\theta}(s_0)$ -- note that this is analogous to minimising $-v_{\pi_\theta}(s_0)$. This is for a stochastic policy. In DDPG the policy is now assumed to be deterministic.
In general, we can write
$$v_\pi(s) = \mathbb{E}_{a\sim\pi}[...
0
Both in DQN and in DDQN, the target network starts as an exact copy of Q-network, that is has the same weights, layers, input_dim, output_dim etc. as the Q-network.
The main idea of the DQN agent is that the Q-network predicts the Q-values of actions from a given state and selects the maximum of them and uses the MSE Loss as its cost function. That is, it ...
2
Am I correct in this understanding that with the increasing complexity of problems, tabular RL methods are getting obsolete?
Individual problems don't get any more complex, but the scope of solvable environments increases due to research and discovery of better or more apt methods.
Using deep RL methods with large neural nets can be a lot less efficient for ...
2
Your policy gradient algorithms appear to be working as intended. All standard MDPs have one or more deterministic optimal solutions, and those are the policies that solvers will converge to. Making any of these policies more random will often reduce their effectiveness, making them sub-optimal. So once consistently good actions are discovered, the learning ...
2
In reinforcement learning (RL), an immediate reward value must be returned after each action, alomng with the next state. This value can be zero though, which will have no direct impact on optimality or setting goals.
Unless you are modifying the reward scheme to try and make an environment easier to learn (sometimes called reward shaping), then you should ...
0
Actually, I just started inspecting the entries further down in the leaderboard list, and there are in fact more modest architectures, e.g. this one, which uses 3 hidden layers with 8 units each.
1
I don't recommend changing the rules of the environment.
What you could do:
Perform a method called bucketing i.e. take a value from a continuous state space see which discrete bucket it should go into and then let your agent use the bucket number as the observation.
e.g. Say I do have a continuous state space with one variable in range $[-\infty,\infty]$
...
2
Lets assume $\sup_{s,a} r(s,a)<b$. Then for continuing problems the upper bound can be obtained by
\begin{align}
\sum_{t=0}^{\infty} \gamma^{t}r(s_t,a_t)
&\le \sum_{t=0}^{\infty} \gamma^{t} \sup_{s,a}r(s,a) \nonumber \\
&=\sum_{t=0}^{\infty} \gamma^{t} b = \frac{b}{1-\gamma}.
\end{align}
We can use the same bound for episodic tasks with ...
2
My answer to:Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem?
Yes but depending on the environment, if dealing with the theoretical environment, where there are infinite number of time steps.
Calculating the upper bound
In reinforcement learning (deep RL inclusive), we want to maximize the discounted ...
2
In any reinforcement learning problem, not just Deep RL, then there is an upper bound for the cumulative reward, provided that the problem is episodic and not continuing.
If the problem is episodic and the rewards are designed such that the problem has a natural ending, i.e. the episode will end regardless of how well the agent does in the environment, then ...
3
To answer your question, the specifics of some of the OpenAI Gym environments can be found on their wiki:
The episode ends when you reach 0.5 position, or if 200 iterations are reached.
There is a deeper question in what you asked, though:
My initial understanding was that an episode should end when the Car reaches the flagpost.
The environment certainly ...
2
The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the _max_episode_steps attribute of the environment.
2
Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
That is the definition of return.
In fact when applying a discount factor this should formally be called discounted return, and not simply "return". Usually the same symbol is used for both ($R$ in your case, $G$ in e.g. Sutton & Barto).
There ...
3
I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural nets come in handy when the state space is very large (or even continuous), so you can't afford to store a table of Q values. Also, neural nets have the ability ...
2
Convergence analysis is about proving that your policy and/or value function converge to some desired value, which is usually the fixed-point of an operator or an extremum. So it essentially proves that theoretically the algorithm achieves the desired function. Without convergence, we have no guarantees that the value function will be accurate or the policy ...
6
This expression: $|\mathcal{A}(s)|$ means
$|\quad|$ the size of
$\mathcal{A}(s)$ the set of actions in state $s$
or more simply the number of actions allowed in the state.
This makes sense in the given formula because $\frac{\epsilon}{|\mathcal{A}(s)|}$ is then the probability of taking each exploratory action in an $\epsilon$-greedy policy. The overall ...
1
The term REINFORCE actually corresponds to a method of estimating gradients, it is not particular to reinforcement learning. The paper you linked doesn't appear to deal with RL at all, so the issue they're describing is not one that you should expect to find in a policy gradient application.
If you're using REINFORCE to estimate policy gradients in RL (this ...
3
In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates.
In both the SARSA and Q learning pseudocode from the book, there is a behaviour policy that selects the next action to take. Other than the initial start state, this ...
0
I think the choice of technique strongly depends on how fine-grained your forecast-predictions need to be.
When it comes to forecasting by Reinforcement Learning (RL), one prominent example is the stock-trading RL agent. The agent must decide which stock to buy or sell, thereby drawing upon predictions concerting the expected future development of some stock....
2
DP, Monte Carlo, and TD are methods of estimating returns. Policy gradient describes methods of learning a policy. So policy gradients serve a different purpose than the other things you mentioned. For clarity, you can use Monte Carlo or TD methods to estimate returns to construct the loss that you get your policy gradient from.
1
As the authors of this paper state it:
In $Q$-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state. This procedure often results in an instability because the values change simultaneously on both sides of the update equation. A target network is a copy of the ...
1
I think there is no simple way to transfer knowledge changes between different models.
If you take your initial model and create a new version of it which you use to learn some other task (like "Walk to a specific location"), then the values copied from the first (original) model change in the second model.
From that moment on, training the former ...
2
If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however that many scientist will often follow the reward hypothesis (Sutton and Barto) which says that
All of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar ...
Top 50 recent answers are included
