1
$\begingroup$

I'm implement PPO for a very specific problem, and it seems to be working somewhat, but after a few epochs, I always get something like this:

/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py:173: UserWarning: Error detected in MulBackward0. Traceback of forward call that caused the error:
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 439, in <module>
    agent.train(120)
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 298, in train
    self.learn(ep=epoch, lr=learning_rate, clip=clip)
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 169, in learn
    clipped_ratio * sampled_normalized_advantage )
 (Triggered internally at  /opt/conda/conda-bld/pytorch_1646756402876/work/torch/csrc/autograd/python_anomaly_mode.cpp:104.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 439, in <module>
    agent.train(120)
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 298, in train
    self.learn(ep=epoch, lr=learning_rate, clip=clip) 
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 197, in learn
    total_loss.backward()
  File "/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'MulBackward0' returned nan values in its 0th output.

I started printing the losses and it turns out that the policy loss becomes nan. I do something like this for the policy loss:

dist, value = self.model(batchStates)
new_probs = dist.log_prob(batchActions)
ratio =  torch.exp(new_probs - batchOldProbs)

clipped_ratio = ratio.clamp(min=1.0 - clip, max=1.0 + clip)
policy_loss = torch.min(ratio * sampled_normalized_advantage,
                   clipped_ratio * sampled_normalized_advantage )
policy_loss = policy_loss.mean()

Should I add a small value to the ratio (1.0e-6 or something), or is it something else?

$\endgroup$
2
  • $\begingroup$ Have you considered taking an approach that does not require exponentiation? Without seeing numbers, that is most likely what is vanishing (as you seem to realize from your question). Could you simply continue to operate on the log values, which will give you a great deal more "room"? $\endgroup$ Jun 2, 2022 at 10:31
  • $\begingroup$ That's how PPO works tho. That's the math behind it. $\endgroup$ Jun 2, 2022 at 10:45

1 Answer 1

3
$\begingroup$

You might want to try substituting the exponentiation with a piecewise-defined function that uses a numerical approximation that is more numerically stable for low values of the exponent, such as using $e^x \approx 1 + x$ (First-order Maclaurin series) when x is under a certain threshold

$\endgroup$
1
  • $\begingroup$ That's actually a very nice trick. I'll accept the answer after some testing. $\endgroup$ Jun 2, 2022 at 11:05

Not the answer you're looking for? Browse other questions tagged .