Consider an environment where there are 2 outcomes (e.g. dead and alive) and a discrete set of actions. For example, a game where the agent has 2 guns $A$ and $B$ to shoot a monster (the monster dies only if the correct gun is used).

Let's say we store the experience $e_1 = (s,a_1, r_1, s'_1)$ in the replay buffer $D$, where

  • $s$: we have the monster to kill
  • $a_1$: choose and use gun $A$
  • $r_1$: $-1$ (the wrong gun)
  • $s'_1$: monster is alive

But we also save the alternative situation $e_2 = (s, a_2, r_2, s'_2)$ in the replay buffer $D$, where

  • $s$: the same state (we have the monster)
  • $a_2$: choose and use gun $B$
  • $r_2$: $(-1) * r = 1$
  • $s'_2$: the monster is dead

I can't find a topic about this technique, or don't know what to look for.

  • $\begingroup$ It looks like for this scenario "if" statement could be the best option. Something like this: "if beast.kind == 1: gans.use.A() else: gans.use.B()". $\endgroup$ Nov 16 '20 at 9:16
  • $\begingroup$ When you ask "What are the implications", what do you mean exactly? Do you mean the "implications in terms of convergence to the right solution during learning/training"? Please, edit your post to clarify this point. Anyway, the closest thing to what you're proposing that I am aware of is Hindsight Experience Replay, but they don't store the alternative tuple, but change the trajectories that are experienced so that they become "falsely successful". $\endgroup$
    – nbro
    Nov 21 '20 at 22:21
  • $\begingroup$ What do you mean by using "IF" statement, the state has +100 features (image and sound), i don't know what should be used "hence the use of the RL". $\endgroup$ Nov 22 '20 at 17:19
  • $\begingroup$ nbro; it's more of a problem of Sample efficiency, and i'm thinking of reversing the states to extend the sample size (as data augmentation). logically it should help, but the loss diverge after a while. $\endgroup$ Nov 22 '20 at 17:25
  • $\begingroup$ @KimbooRasta Please, tag me with @nbro (like I'm tagging you), otherwise I will not see your comments. I saw your comments only by chance. $\endgroup$
    – nbro
    Nov 24 '20 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.