Would it be possible to enforce the same $s_{t + 1}$ between the model's estimate and the target function's Q-value?

Question

Say I have a game of blackjack, and I am trying to teach a single forward-pass neural network to approximate the Q value of the current state and action.

There are 3 inputs: The current card in hand, the cards in the deck, and the cards in the pile. It outputs the Q-value of two actions, namely, holding or adding the current card to the pile.

My loss function is $$L(Q,Q_E)= \sum({Q(s,a_i)- Q_E(s,a_i) )^2},$$

where $Q_E$ is the estimated Q-value of the current state from the policy network. And $Q$ is the target function, which is calculated using the Bellman equation.

As I understand it, the Bellman equation assumes the setting to be deterministic, meaning that, if you're in state $s_t$ as you take action $a_1$, you should always reach the same $s_{t+1}$.

Of course, in blackjack, this is not the case, as the state $s_{t+1}$ is purely dependent on the card you draw, which is a stochastic process.

Would it be possible to omit some of this noise or "stochasticity" by enforcing the same $s_{t + 1}$ between the model's estimate and the target function's Q-value?

In other words, say we're in state $s_t$, and the target function picks action $a_1$ and draws a 10 reaching state $s_{10}$ as the next state. For the training of the policy network, it loads the state $s_t$ from the experience replay, it also picks action $a_1$ and draws a 7 reaching state $s_7$ as the next state.

Would it somehow ruin the training, if I then just hardcoded it, such that the next state reverted to state $s_{10}$ if the policy network picked action $a1$? Are there any counter-productive consequences to this?

Answer 1

As I understand it, the Bellman equation assumes the setting to be deterministic, meaning that, if you're in state $s_t$ as you take action $a_1$, you should always reach the same $s_{t+1}$.

This is not correct, which is a good thing for you. The Bellman equation for action values for an arbitrary policy $\pi(a|s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \mathbf{Pr}\{A_t=a|S_t=s \}$ looks like this:

$$q_\pi(s,a) = \sum_{r,s'}p(r,s'|s,a)(r+\gamma\sum_{a'}\pi(s',a')q_{\pi}(s',a'))$$

The environment model in this equation is $p(r,s'|s,a)$ which is the probability of observing reward $r$ and next state $s'$, given current state $s$ and action $a$. This is stochastic.

When using a model-free approach learning from experience, you cannot use the full equation each time, because you only have one sample and do not know anything about $p()$. However, things will still work, because if you take many samples, on average they will follow the probability distribution for $p()$, and your update rule will converge to the average/expected return for each action value.

The same sampling also applies to the policy, which means the above Bellman equation can be expressed as:

$$q_\pi(s,a) = \mathbb{E}_{A_{t+1} \sim \pi}[R_{t+1} + \gamma q_{\pi}(S_{t+1},A_{t+1})|S_t=s, A_t=a]$$

which in turn means you can use samples of $r, s'$ and $a'$ (or for Q-learning, the maximising $a'$), and an update rule that averages between the values you observe. You can do this with both deterministic and stochastic environments, because the theory behind it works with both.

Would it somehow ruin the training, if I then just hardcoded it, such that the next state reverted to state $s_{10}$ if the policy network picked action $a1$? Are there any counter-productive consequences to this?

For some environments, and some setups of the hardcoding, it may work. However, you should not do this because it is not necessary, and in general it is a bad idea, as it may impact the sampling in a way that causes incorrect learning.