As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just 1 loss value. How does the neural network learn to differentiate between good and bad actions?


How does the neural network learn to differentiate between good and bad actions?

Good actions - in context of a given state - have higher return than bad actions on average, taken over many examples where the actions occur in different combinations.

In REINFORCE, when training the neural network, all actions are effectively treated as ground truth "correct", but the gradient is weighted by the return from that time step. The best trajectories therefore get larger gradient steps, and therefore shift the action choices made in those trajectories more towards them than the less good trajectories.

In the case of positive vs negative returns this is very clear (because negative returns will reverse the gradient step away from the action choices that were made). However, even if all returns are negative or all are positive, the REINFORCE algorithm still works - provided there are enough samples of different trajectories with actions taken in different contexts, then there will be a preference for the best action in each state.

REINFORCE with baseline and Advantage-based update multipliers (which are a variation of REINFORCE with baseline) are slightly better than the most basic REINFORCE "race to the top", in that they are more numerically stable and will automatically split into relative positive and negative updates regardless of whether the environment produces positive or negative returns overall.

Ideally you would not run a REINFORCE update step based on a single trajectory at a time, for two reasons:

  • The trajectory contains correlated data, and neural networks learn badly from it, preferring i.i.d. samples if possible.

  • A single trajectory does not contain enough information by itself for a learning agent to figure out whether it is good or bad relative to other options.

Neither of these are complete showstoppers, but you should find that many implementations of policy gradient methods collect multiple trajectories before each update step.

  • $\begingroup$ I'm having a hard time wrapping my head around this. Each state would have an optimal action(s) compared to other actions. How would REINFORCE learn that by viewing an entire trajectory? While calculating loss/backpropogation, does the algorithm ever compare the prediction for a state with the reward/value obtained? $\endgroup$ Jun 25 at 23:10
  • 1
    $\begingroup$ The algorthm is gradient ascent on the neural network for the output of actions taken times the return measured when taking that action. Unlike classification which rates the action against a known correct value, REINFORCE rates each action according to how well the overall trajectory after that point turned out. Clearly this includes impact of other choices. However, on average each state's choices will favour the best action in the long term. The algorithm never compares anything, it simply follows gradients that produce high returns more than it follows gradients that produce low ones. $\endgroup$ Jun 25 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.