0
$\begingroup$

I use PPO to train my fermenter (digital twin) to maximize enzyme (product) production.

action: 1 or 0 ie. add substrate at a particular time or not based on cell and enzymes present in the tank

observation: enzymes in the tank and timestep

reward : enzyme_slope + cell_slope (Cell slope gives a negative reward due to negative slope as we do not want cells to go negative)

While analysing the training results I saw that the agent was able to find a solution that gave a very high cumulative reward but the agent did not explore in that direction. Whereas when I look at the kind of decisions the agent was taking towards the end of the training I see the cumulative reward is much lower than the max it was able to find before.

The image above clearly gives me more cumulative reward and a higher enzyme activity, agent still does not explore in that direction and decides to explore other things. The final solutions that I am able to find are still good but the I can see that there are better solutions explored by the agent.

Why is this happening? How does PPO decide if a particular solution is good or not and how does it decide to take a certain path? Should I try other algorithms as well?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It looks like the agent hasn't seen the better-rewarding state enough times for its weights to have changed sufficiently enough for it to keep returning to it. Go-Explore proposed to save high-rewarding states and explicitly return to them so that the actor sees them enough times.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Why do you think this happens that it decides to leave a better solution and explore others? $\endgroup$
    – user79474
    Commented Mar 6 at 22:42
  • $\begingroup$ @user79474 Because computers do exactly what you tell them to do. The network you are training has slow-moving features, so of course a single good episode might not be enough. Humans, on the other hand, are able to judge how well their learning is going to intentionally steer themselves in the right direction. We are inspired by how humans think, to steer the training of the network by explicitly returning to high-rewarding states. $\endgroup$
    – Richie Bendall
    Commented Mar 6 at 23:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .