0
$\begingroup$

I'm doing a Deep Q-learning project. All of my rewards are positive and there are two terminal states. One of them has a zero reward and the other has a high positive reward.

The rewards are stochastic and my Q-network must generate non-zero Q-values for all states and actions. Based on my project, I must use these numbers to create a probability density. In other words, the normalized Q-values of each state generated by network define a probability density for choosing an action.

How should I define my loss function? Is there a project or paper which I could look at and decide how to define the loss function? I am searching for similar projects and their proposed Deep Q-learning algorithms.

Improve this question
$\endgroup$
5
  • $\begingroup$ Hi. Please, ask your second question in a different post (I removed it from this post). Also, can you clarify what you mean by "probability density"? Do you mean that you need to estimate the transition probability matrix of the underlying MDP using deep Q learning? If yes, how does this relate to the loss function? $\endgroup$ – nbro♦ Apr 2 '20 at 16:06
  • 1
    $\begingroup$ @nbro: I would guess that "the normalized Q-values of each state generated by network define a probability density for choosing an action" is referring to generating a policy. But it is not clear which policy - e.g. is this a desire to make a behaviour policy based on Boltzmann sampling? That would be relatively normal Q learning. Whilst if this was the target policy i.e. end result from training, then it would be less standard, and the OP would be wanting to learn a non-optimal policy for some unexplained reason. $\endgroup$ – Neil Slater Apr 2 '20 at 16:59
  • $\begingroup$ Also neither of those things should affect loss function choice for the Q values approximation. But that could be part of any answer. It may help if @bitWise explains why they think that the Q network should have a different loss function due to these choices. $\endgroup$ – Neil Slater Apr 2 '20 at 17:02
  • $\begingroup$ @Neil Slater: Q-values generated by the network define the policy of choosing actions. Instead of using an epsilon-greedy policy, I'm using the normalized q-values as the policy of my algorithm. $\endgroup$ – bitWise Apr 2 '20 at 19:43
  • $\begingroup$ @bitWise: Which policy are you wanting to adjust? 1) The behaviour policy to guide exploration and focus more on "reasonable" options . . . . or 2) The target policy which is the final output of the learning process? Both are suitable goals, although the second one is uncommon in Q learning, you could use Expected SARSA off-policy which includes Q learning. $\endgroup$ – Neil Slater Apr 2 '20 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.