0
$\begingroup$

I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to all steps in a rollout.

For this, I do $M$ rollouts, each with $N$ steps, and record the rewards. Let's say I fill an $M \times N$ matrix with the rewards. Sometimes just using these rewards as-is will work, but sometimes the rewards are always positive (or always negative), or the magnitudes cover a large range.

A simple thing is to just subtract the overall mean and divide by the overall standard deviation.

In my particular case, though, the beginning is easier, and during bootstrapping the rewards will be higher. A typical case would have high rewards at the beginning with a taper to zero before the end of the rollout. So, it seems to make sense to subtract the mean along the trial ($M$) dimension. Likewise, it might make sense to normalize the standard deviation along that dimension as well.

My question: Have others already figure this out and developed best practices for normalizing rewards?

I may have added my own twist, but I'm training in batches and using the stats of the batch (multiple rollouts) to do normalization. The subtracting the mean part is called a baseline in some papers I think. This article discusses it a bit.

$\endgroup$
3
  • $\begingroup$ Could you please provide the name of or link to the reference that describes the specific "MC PG" algorithm you're using? $\endgroup$
    – nbro
    Jul 16 at 12:30
  • $\begingroup$ It's REINFORCE, with rewards attributed to individual moves instead of applied to all steps in a rollout. I may have added my own twist, but I'm training in batches and using the stats of the batch (multiple rollouts) to do normalization. The subtracting the mean part is called a baseline in some papers I think. This article discusses it a bit: medium.com/@fork.tree.ai/… $\endgroup$
    – Mastiff
    Jul 16 at 15:27
  • 1
    $\begingroup$ Thanks for clarifying this. I've added the info in your comment to your own post. Feel free to edit your post again if you think it can still be improved. Later, I will delete these comments. $\endgroup$
    – nbro
    Jul 16 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.