As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just 1 loss value. How does the neural network learn to differentiate between good and bad actions?
How does the neural network learn to differentiate between good and bad actions?
Good actions - in context of a given state - have higher return than bad actions on average, taken over many examples where the actions occur in different combinations.
In REINFORCE, when training the neural network, all actions are effectively treated as ground truth "correct", but the gradient is weighted by the return from that time step. The best trajectories therefore get larger gradient steps, and therefore shift the action choices made in those trajectories more towards them than the less good trajectories.
In the case of positive vs negative returns this is very clear (because negative returns will reverse the gradient step away from the action choices that were made). However, even if all returns are negative or all are positive, the REINFORCE algorithm still works - provided there are enough samples of different trajectories with actions taken in different contexts, then there will be a preference for the best action in each state.
REINFORCE with baseline and Advantage-based update multipliers (which are a variation of REINFORCE with baseline) are slightly better than the most basic REINFORCE "race to the top", in that they are more numerically stable and will automatically split into relative positive and negative updates regardless of whether the environment produces positive or negative returns overall.
Ideally you would not run a REINFORCE update step based on a single trajectory at a time, for two reasons:
The trajectory contains correlated data, and neural networks learn badly from it, preferring i.i.d. samples if possible.
A single trajectory does not contain enough information by itself for a learning agent to figure out whether it is good or bad relative to other options.
Neither of these are complete showstoppers, but you should find that many implementations of policy gradient methods collect multiple trajectories before each update step.