What does the Bellman equation actually say? And are there many flavours of that?
I get a little confused when I look for the Bellman equation, because I feel like people are telling slightly different things about what it is. And I think the Bellman Equation is just basic philosophy and you can do whatever you want with that.
The interpretations that I have seen so far:
Let's consider this grid world.
+--------------+ | S6 | S7 | S8 | +----+----+----+ | S3 | S4 | S5 | +----+----+----+ | S0 | S1 | S2 | +----+----+----+
- Rewards: S1:10; S3:10
- Starting Point: S0
- Horizon: 2
- Actions: Up, Down, Left, Right (If an action is not valid because there is no space, you remain in your position)
It tells you how good is it to be in a certain state.
With a horizon of 2, one can reach:
S0==>S3 (Up) (R 5) S0==>S0 (Down) (R 0) S0==>S1 (Right)(R10) S0==>S0 (Left) (R 0)
From that onwards
S0==>S3 (Up) (R 5) S0==>S0 (Down) (R 0) S0==>S1 (Right)(R10) S0==>S0 (Left) (R 0) S1==>S4 (Up) (R 0) S1==>S1 (Down) (R10) S1==>S2 (Right)(R 0) S1==>S0 (Left) (R 0) S3==>S6 (Up) (R 0) S3==>S0 (Down) (R 0) S3==>S3 (Right)(R 5) S3==>S2 (Left) (R10)
Considering no discount, this would mean that it is R=45 good to be in S0, because these are the options. Of course, you can't grab every reward, because you have to decide. Do I need to consider the best next state yet, because this would obviously reduce my expected total reward, but as I can only make two steps it would tell me what is really possible. Not what the overall Reward R(s) in that range is.
This function takes a state and an action, but I am not sure. If that means that I have a reward function that just considers my actions as well to give me a reward. Because in the previous example I just have to land on a state (It doesn't really matter how I get there). But this time I get a reward, when I choose a certain action. R(s,a) But otherwise I do not rate the best action and select that next state to calculate the next state. I choose every next step and from that I choose the 2nd next.
Optimization V-function or Q-function
This works the same as V-Function or Q-Function, but it just considers the next best award. Some sort of greedy approach:
S0==>S3 (Up) (R 5) [x] S0==>S0 (Down) (R 0) [x] S0==>S1 (Right)(R10) S0==>S0 (Left) (R 0) [x]
S1==>S4 (Up) (R 0) [x] S1==>S1 (Down) (R10) S1==>S2 (Right)(R 0) [x] S1==>S0 (Left) (R 0)
So, this would say that is the best I can do in two steps. I know that there is a problem, because when I just follow a greedy approach I risk that I won't get the best result, if I would have had a reward of 1000 on S2 later.
But still, I just want to know, if I have a correct understanding. I know there might be many flavours and interpretations but at least I want to know that is the correct name of these approaches.