# When using (s,a,r,s') to train networks could the Q network be adjusting to a suboptimal r?

My question here is that whenever you take an experience from the experience buffer (s,a,r,s') and you input r into r+yMAXQ(s',a') to get the loss. What if the r from that experience (s,a,r,s') is not the highest possible reward you can get in that state so then when you are calculating the loss and updating the weights isn't it training the network to pick a reward that isn't the maximum possible reward in that state. I realize it is also taking the target estimate into account as well when adjust the weights buy I am talking about specifically the r.

In reinforcement learning, you do not usually wish to train an agent to learn to take actions for maximising the highest possible reward. Instead you wish to train the agent to take actions that maximise the expected reward. An optimal agent is one that, on average, gains the most reward. It is not one that relies on being lucky to score the highest possible reward.

As such, any reward experienced is assumed to be a fair sample of one of the possible rewards received when taking action $$a$$ in state $$s$$. Provided there is no hidden state, and thus the Markov property holds, and the agent samples $$s, a$$ enough times to determine the expected reward from that combination, then this means the agent's estimates will converge towards the true expected action values.

As a simple example, if the reward from taking action $$a_1$$ in state $$s_1$$ was $$-1$$ most of the time, but 5% of the time it was $$+10$$, and the reward from taking action $$a_2$$ in the same state was always $$+1$$, and both action choices transitioned to state $$s_2$$, then action $$a_2$$ should be considered a more optimal choice than action $$a_1$$. Despite the fact that it is possible to get a higher reward in $$a_1$$, the higher expected reward from $$a_2$$ is a better choice.

If the agent is allowed to know the true expected immediated reward from $$s, a$$, the you could use that instead of using the sampled values. You should not use the maximum observed reward so far to update the agent, because that is a biased measure of the expected reward.

• lets say you are in state s and you take action a1 and you get -1 or you can stake action a2 from state s and get +10. Lets say you save the expirience where you take action a1 and get -1 from state s and you save that to the experience buffer. When you got to train the network you get that example from the expirience buffer and input it into the networks and bellman equation. when you calculate the loss and adjust the weights wont that prompt the network to take action a1 more even though it is not the best option in that situation ?
– Stef
Commented Jul 2, 2023 at 22:20
• @Stef In your example, I would expect the network to reduce its estimate of expected return from taking $a_1$ in state $s$. The greedy action at that point is more likely to be $a_2$. However, the whole process is stochastic, and it is possible that the initial estimates for $a_1$ are higher than those for $a_2$ for a few iterations. Once the nwtork has been updated towards targets for $\hat{q}(s,a_1) = -1$ and $\hat{q}(s,a_2) = 10$ after taking several sample of them from memory, then the estimate for taking action $a_2$ will be higher and the agent will prefer it. Commented Jul 3, 2023 at 7:19
• @Stef From your comment, I wonder if you are missing the detail of what the difference between a state value and an action value is? You seem to think learning about the result from action $a_1$ should heavily influence the estimate of the result from $a_2$? In tabular Q learning or SARSA those estimates are entirely separate, and in approximation-based systems (e.g. neural nets), they will still have at least some separate learnable weight parameters where the agent can easily learn the difference Commented Jul 3, 2023 at 7:21
• so when the network learns from experience (s,a,r,s') and you train you model on the experience that gives you -1 as the reward. During training the model is just trying to predict the actual reward of that s a r s' not the best possible action in that state? Does the model take the same action a as in the experience (s,a,r,s') and the try to adjust weights to yield reward r ?
– Stef
Commented Jul 3, 2023 at 15:37
• @Stef It updates the estimate action value for the action that was taken. This is the same in principle as any supervised learning update. The approximation updates the estimate of the output (q) associated with a given input (s,a). To make things a little more complicated, the architecture of the network in DQN might present all actions at once and need some special handling - whether that is important to you depends on whether you are implementing a DQN agent, or using a DQN agent library - the library version will do the necessary handling to only update the specific action being updated. Commented Jul 3, 2023 at 18:30