New answers tagged reinforcement-learning
0
The algorithm can be summarized by the following equation, as done in this post:
$$\mathcal{L}(\theta) = \ell_k \left( r + \gamma . \left(max_{a'~s.t. \frac{G_\omega(a'|s')}{max~\hat{a}~G_\omega(\hat{a}|s')}}{Q_{\theta'}}(s', a') \right) - Q_{\theta}(s, a) \right)$$
Let's imagine that we have the following $Q_{\theta}(s, a)$ for 3 discrete actions: q = np....
2
Whilst engineering solutions in reinforcement learning, I think it is common to discuss the concept of state space loosely, in terms of what the search space looks like for the algorithm, and what compromises are OK even though they technicaly make the problem a POMDP.
In terms of definitions relating to the MDP, the state space has well-defined meaning. It ...
0
I believe you will find the answer in the paper High-Dimensional Continuous Control Using Generalized Advantage Estimation, which is the basis for the advantage function used in the PPO paper that you referenced.
From the paper, the estimate of the advantage function is defined as:
\begin{align*}
\hat{A}_{t}^{GAE(\gamma,\lambda)} = \sum_{l=0}^{\infty}(\gamma\...
2
You may be interested in section 3.2 of this paper What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020) by Google Research.
They claim that the initialization of the policy is very important to performance, sometimes making a huge (66%) improvement, just from the initialization of the policy.
I'm assuming you already know ...
0
You're right, it would not be great to keep training on games played early during the selfplay process, since they would just hurt the network playing strength.
The AlphaZero paper itself does not elaborate, but it has this sneaky sentence:
Unless
otherwise specified, the training and search algorithm and parameters are identical to AlphaGo Zero
The ...
1
I believe that discretizing the action/state space when using function approximators like NN is only acceptable when losing information is acceptable. Why would you discretize an observation, for example, when the precise value of a continuous feature is important for making a decision? Imagine, for example, control scenarios, one of the fields that fit the ...
0
It seems that your description perfectly matches the naive RL approach. What you can do with model-based RL is perform rollouts with the model to predict future states. In other words, with an accurate model you might predict the next state X(t + 1) given the current state X(t) and applying some action a(t). The following (and recently) paper shows how these ...
2
These two algorithms converge to the optimal value function because
they are instances of the generalization policy iteration, so they iteratively perform one policy evaluation (PE) step followed by a policy improvement (PI) step
the PE step is an iterative/numerical implementation of the Bellman expectation operator (BEO) (i.e. it's numerical algorithm ...
1
Generally, "perfect information" is not a formal trait of MDPs. There is a concept of the Markov property, but it only loosely coincides with "perfect information". For instance it is OK for there to be unknown/hidden state, provided it behaves effectively randomly (when revealed, it is drawn from a consistent distribution). An example ...
3
Here's your equation with an additional couple of parenthesis that emphasizes the order of the operations (note that you had a small typo in your original equation).
$$v_{\pi}(s) =\sum_a \pi(a \mid s) \left(\sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$
Now, let me answer your other questions.
Is the second sum using the index $a$ from the ...
1
Intuitively, I feel like if there are 30 foods, each with 2 states, then that is 60 states, no $2^{30}$.
Let's try it with 3 pellets. If you are right there would be $2 \times 3 = 6$ states, if the authors are right there would be $2^3 = 8$ states.
Using * for a pellet, and - for a space, we have the following states:
* * *
* * -
* - *
* - -
- * *
- * -
- -...
1
In the question, you are not describing the environment changing. Instead, there is a fixed 20% chance of a bad weather event each year. Such events can me modelled as a static environment with stochastic results.*
If nothing else happened in the year, it is easy to calculate the expected immediate reward for each action:
Not planting seeds. $0$
Planting ...
1
Predicting the correct amount of repetitions for an action sounds like a regression task. Turning it into a classification task using a model with n output nodes will lead to several drawbacks, the biggest ones being:
Having to choose a priori a finite max amount of actions n
Turning the data into really sparse vectors, especially for large n.
So a better ...
2
You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do not always take the action with the highest probability (otherwise there would be no exploration but just exploitation), but the most probable action should be sampled the most. However, keep in mind that the policy, $\theta$, changes, so also the probability distribution....
2
Adding to Neil's reply, though the path shown is optimal, following the so-called 'optimal path' will often result in sub-optimal returns because the action selection in this problem is stochastic due to the $\epsilon$-greedy exploration. That is, even though if we are in a block right above the cliff region and know that the best action to do is to move ...
4
It is important to note that the graph shows reward received during training. This includes rewards due to exploratory moves, which sometimes involve the agent falling off the cliff, even if it has already established that will lead to a large penalty. Q-learning does this more often than SARSA because Q learning targets learning values of the optimal greedy ...
1
there should be absolutely no problem with training an agent on any available episode roll-out data. That is because a MDP implies for an any state S, the optimal action to take is entirely dependent on the state.
The desired end-state of the trained model is that it can identify the optimal action. When comparing reinforcement learning (RL) methods, you ...
1
I think that your description is roughly correct, but I wouldn't call a "sampling model" a "model" because it doesn't necessarily model something, unless, for example, you are first learning in simulation to later be able to act in the real-world or environment (in this sense, the simulation would be a model of the real environment, but ...
1
What you did is incorrect and that's not what authors got either (if you're refering to the equation above equation (5) in paper "Non-parametric Policy Search with Limited Information Loss")
What you need here is the derivative of a functional. Functional has a general form
\begin{equation}
J(f) = \int_x L(x, f(x), f'(x), \ldots, f^{(n)}(x)) dx
\...
Top 50 recent answers are included