UPDATE: After reading more about the topic, I've tried implementing the DDPG algorithm instead of using a variation of Q-Learning and still have the same issue.
I have the following issue:
I want to train my critic to estimate values of state/action pairs. My state consists of 2 real valued variables and my action is another real valued variable.
I normalize all values before I feed them into the network. Now I have the following issue: The network is very unresponsive to changes in the input. Before I normalize the state and the actions, they can take up any value between 0 and 50. After normalizing them they are in the range between -1 and 1. A change of 1 in the input can become a very small change in the input after normalization.
But in my specific situation, a small change in the action or the state can cause a very large change in the value of the state/action pair. The network does not really learn that correctly, it handles similar inputs similarly all the time (which is okay most of the times, but there are hard cuts in the shape of the value function here and there). If I further reduce the networks' capacity, the networks output becomes constant and ignores all three inputs.
Do you know any other tricks, that I could use to increase sensitivity to the input at some points? Or is my network configuration/approach the wrong one (too large, too small)?
The network I'm training is a simple feedforward neural network that takes two inputs, followed by 2 hidden layers followed by a single output to predict the value for that state/action combination. (I'm still trying out different configurations here, as I have no real feeling for the amount of elements per layer and the amount of layers needed to get the capacity I need without encouraging overfitting).
Thanks for your help :)