0
$\begingroup$

I'm wondering whether problems with a custom PPO implementation (problem couldn't be made into a gym environment) can be debugged considering how the losses change over time. In my current experiment, although results do get better in the actual task, the actor(policy) loss, the value function loss and the entropy move in weird ways. enter image description here

enter image description here

enter image description here

How do these metrics usually change over the course of an experiment? Do they always move along the same paths? Thank you.

$\endgroup$

1 Answer 1

1
$\begingroup$

It's difficult to tell how the output of the loss changes over the course of the experiment as you use them as a measure on how well your model performs. Ideally, the loss decreases over time with minimal jittering. A decreasing loss means the model is learning and the hyper parameters fit the dataset.

If your loss is jittering too much without a clear decreasing trend, it might be that, for example, the learning rate is too high and the model overshoots the minimum. It also might be that there is a flaw in the model.

To make it short, no, the loss doesn't always move along the same path.

If you need to test your metric, create tests with small models. Use a small keras model (for example) where you know that there is no flaw inherent with the model and use standard hyper parameters. Then train the model twice. Once with your own metric and once with a standard metric similar to your own provided by a library. The path of the loss should look similar - but it will not be the same.

I'd do the above approach for every method you implemented yourself to make sure that the method itself is doing what it is intended to do.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .