1
$\begingroup$

I am studying Q-learning in reinforcement learning. My question is about the Bellman equation.

In Q-learning, the Bellman equation is often introduced as follows.

\begin{align} Q_{new}(s,a) &= Q_{old}(s,a) + \text{learning rate} \times \text{error}\\ &= Q_{old}(s,a) + \alpha(\text{target} - \text{actual})\\ &= Q_{old}(s,a) + \alpha((\text{reward} + \text{discount factor} \times \text{max next } Q) - Q_{old}(s,a)) \\ &=Q_{old}(s,a) + \alpha[r(s,a)+\gamma\times max(Q(s',a'))-Q_{old}(s,a)] \end{align}

The update equation of gradient descent (which is used in the context of neural networks and other fields) is as follows.

$$ w_{new} = w_{old} + \eta\frac{dE}{dw} $$ So, why the Bellman equation depends on the error after the learning rate while the gradient descent depends on the error gradient? I feel confused.

$\endgroup$
4
  • 2
    $\begingroup$ I provided an answer to the general question in the title. However, I don't fully understand the specific question in the body of your post. In particular, I don't understand this "Bellman equation depends on the error after the learning rate". What do you mean by "error after the learning rate"? If you're asking whether the difference between the Bellman equation and the GD rule is that one uses the "error" while the other uses the "gradient", then the answer is "yes". $\endgroup$
    – nbro
    Sep 29 at 14:20
  • 1
    $\begingroup$ Yes, this is what I mean. I know this is a difference. But why? In other words, why the Bellman equation does not use the gradient instead of error? I think this may be mathematically inappropriate $\endgroup$ Sep 30 at 3:57
  • $\begingroup$ If you're wondering why Q-learning (or TD-learning) are defined using a Bellman equation that uses the "temporal difference" and why it works at all, you should probably ask a different question in a separate post that doesn't involve gradient descent. It seems to me that you know the main difference between GD and TD learning, although you are asking that question in the title and that's the question that I answered. It seems to me that your doubt is more "Why does Q-learning even work using this type of update rule?". $\endgroup$
    – nbro
    Sep 30 at 16:02
  • $\begingroup$ In this post, you were effectively asking 2 quite distinct questions. Ideally, one post should contain exactly one question, as explained here. $\endgroup$
    – nbro
    Sep 30 at 16:03
2
$\begingroup$

They have a few similarities, but they are quite different. Let me first give you a general description of both approaches/algorithms, so that you start to get a sense of their differences and similarities.

Description

Gradient descent (GD) can be applied to solving any optimization problem where your loss (aka cost or objective) function is differentiable with respect to the parameters that you want to update. For example, if you're training a neural network with gradient descent to solve a classification problem, you could be using a cross-entropy function, which should be differentiable with respect to the weights of the neural network (so you should make sure that all the operations in the neural network, in particular, the activation functions, are differentiable or, at least, you define their derivatives). So, the only restriction to use GD is that your loss function is differentiable, so you could use GD to solve classification, regression, or even reinforcement learning problems (and this has actually be done).

Temporal-difference (TD) learning is a specific approach to reinforcement learning, where you update your current estimate of a value function (in your case, the action, aka state-action, value function) with a value that is the difference between estimates at different time steps (hence the name temporal difference).

How are they different/similar?

  • They can both be seen as learning algorithms/approaches, although people in other areas other than machine learning may view gradient descent "just" as an optimization algorithm.

  • TD learning is applied in the specific context of reinforcement learning, while GD is applied to any optimization problem where your cost function is differentiable.

  • In TD learning (or, more generally, in RL), you want to find a value function (or policy), which could be seen as the parameters that we want to find. (So, here, the parameters are the variables that we want to find). On the other hand, in GD, we want to find the parameters of a model (that define some function or distribution, which could be a value function, but not necessarily).

  • TD learning can be combined with neural networks (for example, see this paper), which leads to a new field often known as deep RL. In this case, you may use GD to update the parameters of this neural network, which represents the value function or policy. So, GD can be used to solve RL problems

  • In both cases, we have a learning rate, which determines the magnitude of the changes to the current estimate of the parameters.

  • You can estimate/approximate gradients (or derivatives) with finite-differences, or finite-differences could be seen as the discrete version of derivatives. In fact, derivatives can be defined as limits of differences. Moreover, if you read e.g. this paper, you will see a lot of gradient symbols. Given that TD uses these "differences", this could be the reason why you're confused.

$\endgroup$
1
  • $\begingroup$ Thank you very much for this detailed answer $\endgroup$ Sep 30 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.