Consider an environment where there are 2 outcomes (e.g. dead and alive) and a discrete set of actions. For example, a game where the agent has 2 guns $A$ and $B$ to shoot a monster (the monster dies only if the correct gun is used).
Let's say we store the experience $e_1 = (s,a_1, r_1, s'_1)$ in the replay buffer $D$, where
- $s$: we have the monster to kill
- $a_1$: choose and use gun $A$
- $r_1$: $-1$ (the wrong gun)
- $s'_1$: monster is alive
But we also save the alternative situation $e_2 = (s, a_2, r_2, s'_2)$ in the replay buffer $D$, where
- $s$: the same state (we have the monster)
- $a_2$: choose and use gun $B$
- $r_2$: $(-1) * r = 1$
- $s'_2$: the monster is dead
I can't find a topic about this technique, or don't know what to look for.