New answers tagged policy-gradients
1
There is no sign error and we should not change to $\arg\max$. With Policy Gradients I find that it is not useful to think about things such as a 'loss'.
In short, we want to first find the derivative of the RL objective $J(\theta) = v_\pi(s_0)$, where $\pi$ is our policy that depends on some parameters $\theta$. The policy gradient theorem tells us that
$$\...
1
They are not maximizing the gradient, the gradient is of the form
\begin{equation}
\nabla_{\theta} J \approx \sum_{t=0}^T G_t \nabla_{\theta} \log(\pi_{\theta}(a_t|s_t))
\end{equation}
that means that when implementing it in software you can form your objective as
\begin{equation}
J = \sum_{t=0}^T G_t \log(\pi_{\theta}(a_t|s_t))
\end{equation}
and then ...
2
generally the approach is to have a separate head. For example, imagine you have latent vector $z_k$, you would output two values: $h(z_k)$ and $f(z_k)$ where $0 \leq h \leq 1$ and $b_0 \leq f \leq b_1$ where $b_0$ and $b_1$ are your bounds.
In thios setup, during inference you would check $h_k$ and if its greater than some threshold (usually .5), youd ...
Top 50 recent answers are included
Related Tags
policy-gradients × 134reinforcement-learning × 119
actor-critic-methods × 16
neural-networks × 15
reinforce × 15
proximal-policy-optimization × 14
deep-rl × 13
ddpg × 13
machine-learning × 7
comparison × 7
off-policy-methods × 7
python × 6
loss-functions × 6
gradient-descent × 6
rewards × 6
deep-learning × 5
math × 5
pytorch × 5
q-learning × 4
backpropagation × 4
proofs × 4
policies × 4
experience-replay × 4
trust-region-policy-optimization × 4
training × 3