UPDATE: After reading more about the topic, I've tried implementing the DDPG algorithm instead of using a variation of Q-Learning and still have the same issue.

I have the following issue:

I want to train my critic to estimate values of state/action pairs. My state consists of 2 real valued variables and my action is another real valued variable.

I normalize all values before I feed them into the network. Now I have the following issue: The network is very unresponsive to changes in the input. Before I normalize the state and the actions, they can take up any value between 0 and 50. After normalizing them they are in the range between -1 and 1. A change of 1 in the input can become a very small change in the input after normalization.

But in my specific situation, a small change in the action or the state can cause a very large change in the value of the state/action pair. The network does not really learn that correctly, it handles similar inputs similarly all the time (which is okay most of the times, but there are hard cuts in the shape of the value function here and there). If I further reduce the networks' capacity, the networks output becomes constant and ignores all three inputs.

Do you know any other tricks, that I could use to increase sensitivity to the input at some points? Or is my network configuration/approach the wrong one (too large, too small)?

The network I'm training is a simple feedforward neural network that takes two inputs, followed by 2 hidden layers followed by a single output to predict the value for that state/action combination. (I'm still trying out different configurations here, as I have no real feeling for the amount of elements per layer and the amount of layers needed to get the capacity I need without encouraging overfitting).

Thanks for your help :)

  • $\begingroup$ It usually means that network doesn't have an adequate complexity to fit the wanted function. But I'm curious how do you even do the q-learning update in continuous action space, how do you calculate $max_{a'} Q(s, a')$ ? Policy gradient / actor-critic methods are more appropriate for cases when you have continuous action space $\endgroup$ – Brale Jul 11 '19 at 16:06
  • $\begingroup$ Thanks for the tip with policy gradients. I'm fresh to RL, currently trying to implement my system using DDPG. I've calculated it until now by making a guess and sampling, which basically turns it back into a discrete action space. $\endgroup$ – Raandom Jul 24 '19 at 12:42
  • $\begingroup$ Just to have it said: I still have exactly the same problem, only with the critic now. As the critic is in fact learning more or less the same function I have learned before (as it tries to estimate the value of the state/action combination), it still does not learn the actual distributions. @Brale_ $\endgroup$ – Raandom Jul 24 '19 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.