# Questions tagged [bellman-equations]

For questions related to the Bellman equations in the context of reinforcement learning (and other artificial intelligence subfields).

40 questions
Filter by
Sorted by
Tagged with
8k views

### What is the Bellman operator in reinforcement learning?

In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as a function where the ...
198 views

### Why state-action value function as an expected value of the return and state value function, does not need to follow policy?

I often see, the state-action value function is expressed as: $q_{\pi}(s,a)=\mathbb{E}_{\pi}[R_{t+1}+\gamma G_{t+1} | S_t=s, A_t = a] = \mathbb{E}[R_{t+1}+\gamma v_{\pi}(s') |S_t = s, A_t =a]$ Why ...
973 views

### What is the proof that policy evaluation converges to the optimal solution?

Although I know how the algorithm of iterative policy evaluation using dynamic programming works, I am having a hard time realizing how it actually converges. It appeals to intuition that, with each ...
215 views

### How is the DQN loss derived from (or theoretically motivated by) the Bellman equation, and how is it related to the Q-learning update?

I'm doing a project on Reinforcement Learning. I programmed an agent that uses DDQN. There are a lot of tutorials on that, so the code implementation was not that hard. However, I have problems ...
232 views

### Why do Bellman equations indirectly create a policy?

I was watching a lecture on policy gradients and Bellman equations. And they say that a Bellman equation indirectly creates a policy, while the policy gradient directly learns a policy. Why is this?
175 views

60 views

### Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?

I was looking at the Bellman equation, and I noticed a difference between the equations used in policy evaluation and value iteration. In policy evaluation, there was the presence of $\pi(a \mid s)$, ...
137 views

### Equation not satisfied in Policy Iteration Algorithm

In equation 4.9 of Sutton and Barto's book on page 79, we have(for policy iteration algo): $\pi ^{'}(s) = arg \max_{a}\sum_{s',r}p(s',r|s,a)[r+\gamma v_{\pi}(s')]$ where $\pi$ is the previous policy ...
111 views

### Are these two definitions of the state-action value function equivalent?

I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function ...
329 views