I am training a trading bot with TD3 and SAC algorithms. During the first 10k steps it takes uniformly random actions before running policy learnt so far. The agent starts to do gradient descent updates after the first 3k steps. For some reason after the initial exploration, i.e. first 10k steps (~26 episodes), episode returns drop, see below.
I changed learning rate, number of units in the layers, mini-batch size, added more layers but with no success. Has anyone any idea what is going on?