Why might my policy gradient agent appear to maximize the absolute value of rewards?

Question

I have a toy policy gradient RL algorithm using REINFORCE (aka monte carlo policy gradients) that involves bots moving on a grid attempting to "acquire" targets in Pytorch. The bots receive +1 for moving closer to targets, -1 for moving farther away/invalid actions, some larger amount for acquiring a target (a rare action), and a smaller penalty for inaction.

What I have observed is that the agents swiftly learn to attempt moving off the grid. In doing so, the bot has learned to ironically maximize the negative reward.

So I tried swapping the sign of all the rewards, and yet the behavior persists, hence my belief that it's almost learning on the absolute value of the reward. It's also possible the network is just randomly converging to one action, but it's unclear how this is the case if so.

Why might this be the case?

My code is long and the environment self-defined, but I believe the relevant parts should be my model action choice and the reinforce part. Rewards and movement themselves seem fine as, if I comment out the reinforcement updated, rewards and movement are applied as expected to the random outcomes of the network.

Here's the model output part:

    def forward(self, x):

      # Run through all of our layers defined above
      x = #...removed for brevity
      x = F.softmax(self.linear_final(x), dim=1)

      return x

    def decide_action(self, x):

      # Create the prob distribution from output and return the action/logprob
      probs = self.forward(x)
      prob_dist = Categorical(probs)
      action = prob_dist.sample()
      action_item = action.item()
      logp = prob_dist.log_prob(action)

      return action_item, logp

Here's the reinforce part:

# Function to calculate loss and update bot network
def reinforce_bot(b, debug):

    # Setup lists and vars to work off
    discounted_reward = 0
    l_returns = []
    l_policy_loss = []

    # Work through the bot's episode rewards backwards
    # The net effect of this will be such that we built rewards for only actions and their following rewards
    # (i.e. action for step n only gets rewards for steps > n, never steps < n)
    # Additionally we'll build in our reward discounting (where future steps contribute less to overall reward)
    for reward in b.l_episode_rewards[::-1]:
        discounted_reward = reward + gamma * discounted_reward
        l_returns.insert(0, discounted_reward) # but insert back at the beginning to get correct order
    
    # Now turn the rewards into a tensor for working with gradient
    t_returns = torch.tensor(l_returns)
    # But standardize the rewards to stabilize training
    t_returns = (t_returns - t_returns.mean()) / (t_returns.std())

    # Now build up our actual policy loss by multiplying it by our logprobs
    for logp, discounted_reward in zip(b.l_episode_log_probs, l_returns):
        l_policy_loss.append(-logp * discounted_reward)

    # Zero our gradient to get ready for backprop
    b.optimizer.zero_grad()

    # Technically our l_policy_loss is a list of tensors, so smoosh those together
    # Then sum to get the total loss
    policy_loss = torch.cat(l_policy_loss).sum()

    # Now run our optimizer
    policy_loss.backward()
    b.optimizer.step()

    # Text to print some helpful debugging
    if debug:
        print(f'{b.team} had awards array of {b.l_episode_rewards}')
        print(f'{b.team} had l_returns of {l_returns}')
        #print(f'{b.team} had l_policy_loss of {l_policy_loss}')
        print(f'{b.team} had a policy loss of  {policy_loss}')

    # And cleanup our epsiode tracking lists now since we don't need them
    del b.l_episode_rewards[:]
    del b.l_episode_log_probs[:]

Perhaps the implementation of REINFORCE is not properly handling the mix of positive and negative rewards?

Answer 1

I've found at least part of the problem -- quite simply the learning rate was too high at 0.01. My guess is that, since rewards are fairly common, the agent would occasionally find cases where heading in a certain direction constantly gives it a reward, the gradient explodes, and the agent is unable to recover. The seen behavior then is eventually an agent that, in a new context, is repeating a move that reaps negative rewards (but is unable to properly update it's way out of it).

I also implemented a different version of reward discounting that looks more like reward * gamma ** i which causes the rewards to exponentially decay over the array. Convergence with this was much quicker and more stable, but it's still unclear to me if Pytorch's official example has issues or what is really appropriate for rewards processing.