# What are the popular approaches to estimating the Q-function?

I need the q-value for my RL training, there are some approaches:

• Brute-force the action sequence (this won't work for long sequence)
• Use a classic algorithm to optimise and estimate (this ain't much AI)
• Create Monte Carlo samples and train an approximator network for calculating q-value

I find the Monte Carlo method above rather widely applicable to different problems, and the more computing power, the more precise it is. Any other methods for calculating q-value?

• How much RL do you know? When you say classic algorithms do you mean Dynamic Programming algorithms? – David Ireland Feb 1 at 13:30
• anything, not just dynamic programming. i can use some kind of greedy and random optimisation for the action sequence after the action in q(s,a); that's what i meant about 'classic algo' – datdinhquoc Feb 1 at 13:51
• Do you know any existing RL methods, such as Q-learning, SARSA? If I knew your existing knowledge of RL it would be easier to answer the question. – David Ireland Feb 1 at 14:59
• @datdinhquoc To me, this question is not fully clear (e.g. I don't understand what you mean by "Brute-force the action sequence"), so I agree with David that you should describe what you know about RL, and why you're trying to estimate a Q-table without apparently wanting to use RL algorithms, such as Q-learning. When we use "approximation" in RL we typically mean function approximation, but it doesn't seem that you meant that. You just want to "estimate" the state-action value function, apparently. So, what is your question? Are you asking which algorithms are there to estimate Q(s, a)? – nbro Feb 1 at 17:36
• @datdinhquoc Why don't you simply use Q-learning? – nbro Feb 2 at 18:56

There's are some solutions to calculating q-values; find the exact values:

• Brute-force the action sequence to find max (not pratical)
• Do recursion on Bellman equation to get max (the same like action sequence brute-force, not pratical)

Estimate the q-values:

• Based on different problems to solve, apply some classic algorithms or human logics to estimate; during estimation, some heuristic tactics may be used
• Do a lot of randomisation to find max, including Monte Carlo tree search

(1) Do optimisation based on Bellman equation (Q-learning): $$q_t(s_t,a_t) = q_t(s_t,a_t) + \alpha(r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1})) - q_t(s_t,a_t))$$
Bellman equation is true when the temporal difference (the part multiplied with $$\alpha$$) reaches zero, which means the max of time t+1 reaches exact value.
(2) Do optimisation based on Bellman equation (Q-network), fit the neural network to expected value: $$r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1}))$$