I need the q-value for my RL training, there are some approaches:

  • Brute-force the action sequence (this won't work for long sequence)
  • Use a classic algorithm to optimise and estimate (this ain't much AI)
  • Create Monte Carlo samples and train an approximator network for calculating q-value

I find the Monte Carlo method above rather widely applicable to different problems, and the more computing power, the more precise it is. Any other methods for calculating q-value?

  • 1
    $\begingroup$ How much RL do you know? When you say classic algorithms do you mean Dynamic Programming algorithms? $\endgroup$ – David Ireland Feb 1 at 13:30
  • $\begingroup$ anything, not just dynamic programming. i can use some kind of greedy and random optimisation for the action sequence after the action in q(s,a); that's what i meant about 'classic algo' $\endgroup$ – datdinhquoc Feb 1 at 13:51
  • 1
    $\begingroup$ Do you know any existing RL methods, such as Q-learning, SARSA? If I knew your existing knowledge of RL it would be easier to answer the question. $\endgroup$ – David Ireland Feb 1 at 14:59
  • 2
    $\begingroup$ @datdinhquoc To me, this question is not fully clear (e.g. I don't understand what you mean by "Brute-force the action sequence"), so I agree with David that you should describe what you know about RL, and why you're trying to estimate a Q-table without apparently wanting to use RL algorithms, such as Q-learning. When we use "approximation" in RL we typically mean function approximation, but it doesn't seem that you meant that. You just want to "estimate" the state-action value function, apparently. So, what is your question? Are you asking which algorithms are there to estimate Q(s, a)? $\endgroup$ – nbro Feb 1 at 17:36
  • 2
    $\begingroup$ @datdinhquoc Why don't you simply use Q-learning? $\endgroup$ – nbro Feb 2 at 18:56

There's are some solutions to calculating q-values; find the exact values:

  • Brute-force the action sequence to find max (not pratical)
  • Do recursion on Bellman equation to get max (the same like action sequence brute-force, not pratical)

Estimate the q-values:

  • Based on different problems to solve, apply some classic algorithms or human logics to estimate; during estimation, some heuristic tactics may be used
  • Do a lot of randomisation to find max, including Monte Carlo tree search

Gradually optimise the q-values:

(1) Do optimisation based on Bellman equation (Q-learning): $$ q_t(s_t,a_t) = q_t(s_t,a_t) + \alpha(r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1})) - q_t(s_t,a_t))$$

Bellman equation is true when the temporal difference (the part multiplied with $\alpha$) reaches zero, which means the max of time t+1 reaches exact value.

(2) Do optimisation based on Bellman equation (Q-network), fit the neural network to expected value: $$r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1}))$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.