I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per episode which I can compute after the last action of the episode.
For each timestep (~ action) my agent has 20 choices. The following plot shows the reward (y-axis) versus the current episode (x-axis). The plot shows a decreasing reward because I want to mimize this reward so I use: minus of the true reward.
At the beginning of the process, the agent seems to learn very well and the reward is decreasing but then it's converging to a value which is not the best. When I look at the results of my experiment it appears that the index of all actions are same (for example the agent is always choosing the second value of my discrete action space).
Does anyone have an idea about what is happening here ?