# What are the implications of storing the alternative situation (that could have been experienced) in the replay buffer?

Consider an environment where there are 2 outcomes (e.g. dead and alive) and a discrete set of actions. For example, a game where the agent has 2 guns $$A$$ and $$B$$ to shoot a monster (the monster dies only if the correct gun is used).

Let's say we store the experience $$e_1 = (s,a_1, r_1, s'_1)$$ in the replay buffer $$D$$, where

• $$s$$: we have the monster to kill
• $$a_1$$: choose and use gun $$A$$
• $$r_1$$: $$-1$$ (the wrong gun)
• $$s'_1$$: monster is alive

But we also save the alternative situation $$e_2 = (s, a_2, r_2, s'_2)$$ in the replay buffer $$D$$, where

• $$s$$: the same state (we have the monster)
• $$a_2$$: choose and use gun $$B$$
• $$r_2$$: $$(-1) * r = 1$$
• $$s'_2$$: the monster is dead

• It looks like for this scenario "if" statement could be the best option. Something like this: "if beast.kind == 1: gans.use.A() else: gans.use.B()". Nov 16 '20 at 9:16
• When you ask "What are the implications", what do you mean exactly? Do you mean the "implications in terms of convergence to the right solution during learning/training"? Please, edit your post to clarify this point. Anyway, the closest thing to what you're proposing that I am aware of is Hindsight Experience Replay, but they don't store the alternative tuple, but change the trajectories that are experienced so that they become "falsely successful".
– nbro
Nov 21 '20 at 22:21
• What do you mean by using "IF" statement, the state has +100 features (image and sound), i don't know what should be used "hence the use of the RL". Nov 22 '20 at 17:19
• nbro; it's more of a problem of Sample efficiency, and i'm thinking of reversing the states to extend the sample size (as data augmentation). logically it should help, but the loss diverge after a while. Nov 22 '20 at 17:25
• @KimbooRasta Please, tag me with @nbro (like I'm tagging you), otherwise I will not see your comments. I saw your comments only by chance.
– nbro
Nov 24 '20 at 14:46