Let's say we've a given policy whose value function is to be evaluated. One way to get the value function can be using expected SARSA, as in this stack exchange answer. However, my MDP's state space is massive, so I am using a modified version of DQN that I call deep expected SARSA. The only change from DQN is that the target policy is changed from 'greedy wrt. value network' to 'the given policy' whose value is to be evaluated.

Now on training a value function using deep expected SARSA, the loss curve that I see don't show a decreasing trend. I've also read online that DQN loss curves needn't show decreasing trend and can be increasing and it's okay. In this case, if loss curve isn't necessarily going to show decreasing trend, how do I measure the accuracy of my learned value function? Only idea I have is to compare output of learned value function at (s,a) with expected return estimated from averaging returns from many rollouts starting from (s,a) and following given policy.

I've two questions at this point

  1. Is there a better way to learn the value function than deep expected SARSA? Couldn't find anything in literature that did this.
  2. Is there a better to way to measure accuracy of learned value function? Thank you very much for your time!


You must log in to answer this question.