3

How does the neural network learn to differentiate between good and bad actions? Good actions - in context of a given state - have higher return than bad actions on average, taken over many examples where the actions occur in different combinations. In REINFORCE, when training the neural network, all actions are effectively treated as ground truth "...


2

An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the mechanism to handle invalid actions. However, those main references are still only pre-prints in the arxiv (not published and presumably not peer-reviewed yet). ...


2

You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do not always take the action with the highest probability (otherwise there would be no exploration but just exploitation), but the most probable action should be sampled the most. However, keep in mind that the policy, $\theta$, changes, so also the probability distribution....


2

The policy gradient states that $$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a q_\pi(s, a) \nabla\pi(a | s; \theta)\;$$ where the derivatives are taken wrt the parameter $\theta$. Now, if we say incorporate a baseline we get $$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a \left( q_\pi(s, a) - b(s) \right)\nabla\pi(a | s; \theta)\;$$ and this does not effect ...


1

They are not maximizing the gradient, the gradient is of the form \begin{equation} \nabla_{\theta} J \approx \sum_{t=0}^T G_t \nabla_{\theta} \log(\pi_{\theta}(a_t|s_t)) \end{equation} that means that when implementing it in software you can form your objective as \begin{equation} J = \sum_{t=0}^T G_t \log(\pi_{\theta}(a_t|s_t)) \end{equation} and then ...


1

When the authors write go from $$\nabla_{\theta}J \propto \sum_s \mu(s) \sum_a q_{\pi}(s,a)\nabla_{\theta}\pi(a|s;\theta)\;$$ to $$\nabla_{\theta}J = E_{\pi}\left[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t;\theta)\right]\;$$ they are simply taking an expectation where the only random variable is the state $S_t$. This is because, as they say in the book, ...


1

The fundamental idea behind policy gradient is just to maximise the return averaged across all probably trajectories, i.e $$\begin{align} J(\theta) &= E[\sum\limits_{t=1}^{\tau}r(s_t,a_t)]\\ &=E_{\tau\sim p(\tau)}[R(\tau)] \end{align}$$ Where $\tau$ represents the probability of selecting a particular trajectory, if the trajectories all have fixed ...


Only top voted, non community-wiki answers of a minimum length are eligible