PPO agent. The action space includes 3 actions:
- 0: do nothing
- 1: act (start)
- 2: stop
The agent has to perform thousands of steps doing nothing, then perform step 1 only once (act), then do nothing for N steps again (i.e. observe) for some time, then trigger action 2. Action 2 ends the episode.
An example might be fishing: wait till good state conditions, cast a fishing rod, wait till it's time, pull the fish (or nothing). I give small negative rewards for waiting before casting, and explicit feedback reward after casting so the agent knows how it is doing.
Problem: after 50K times of training, the agent decides that collecting negative rewards and doing nothing is 100% better than doing anything else, even though rewards would be higher.
Tweaking rewards, including more severe punishment for doing nothing before acting and bigger rewards for acting and stopping, doesn't help.
My guess is that it is caused by the fact that we have a disproportionately big number of cases when we do nothing, i.e. actions imbalance. Is there a way to deal with this kind of problem?
If you think it is not the root cause, I'd highly appreciate other suggestions.
(P.S. please ignore the episode reward decline below -25; it is caused by increasing the
number of steps before acting the agent is allowed to use during an episode)