Actor loss in the DDPG algorithm is:
critic_value = critic_model([state_batch, action_batch]) actor_loss = -tf.math.reduce_mean(critic_value)
As I was trying to tune the algorithm, I noticed that the moving average of actor loss has some relationship with buffer size. With a buffer size of 100000, the moving average is ascending. With a size of 400000, the moving average will be descending. And with 200000, it would be almost steady.
I should note that in my problem rewards of every step are positive numbers and the step of every episode is fixed! For example, after 4 steps the episode will be over. Therefore, it's different from the problems that try to have longer episodes.
What is the relationship between buffer size and actor loss that causes this? What buffer size should I choose then? Is the moving average of actor loss supposed to be ascending, descending, or steady? Is there a problem with my target network that would cause this??