1
$\begingroup$

I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the threshold.

My question is: should I select state-action pairs by their immediate reward or by the episode reward?

I am applying the method to a craft environment interesting to me and I have been studying an example applying the OpenAI's Gym taxi environment, but I do not fully understand the code.

Improve this question
$\endgroup$
6
  • $\begingroup$ Hello. I've edited your post in order to clarify what your question is by putting it in the title. Please, make sure that the current version of your post is still consistent with the original one. If not, feel free to edit your post again to fix that. $\endgroup$
    – nbro
    Jul 22, 2021 at 10:39
  • $\begingroup$ Although the edition is Ok, I am not sure if monte-carlo-methods as tag is correct. $\endgroup$
    – Hermes Morales
    Jul 22, 2021 at 11:58
  • 1
    $\begingroup$ It is Monte Carlo in the general sense of simulating outcomes by sampling from random variables (in this case only the policy). I would not have used monte-carlo-methods here, CEM is not a variant of Monte Carlo control. Although you could consider REINFORCE algorithm strongly related to CEM, and an "upgrade" of it. $\endgroup$
    – Neil Slater
    Jul 22, 2021 at 12:36
  • $\begingroup$ It may be a good idea to provide a reference/link to the CEM you're referring to, just to make sure we're talking about the same thing. $\endgroup$
    – nbro
    Jul 22, 2021 at 14:47
  • $\begingroup$ @NeilSlater I am refining my skills related to Reinforcement Learning after doing some online courses in Coursera. I intend to apply these knowledge area to the adaptation of livestock farmers to climate variability. As an excercise, I am trying to deep my expertise in the different kind of RL and related algorithms. For that, initially I must master the environment construction under different perspectives, tabular and approximation methods, discrete and continuous state and action spaces, and solidify my familiarity with the discipline $\endgroup$
    – Hermes Morales
    Jul 22, 2021 at 16:56

1 Answer 1

Reset to default
1
$\begingroup$

My question is if I should select state_action pairs by theirs immediate reward or should I select them by the episode reward?

By the return (sum of all rewards) from the whole episode. A lot of decisions made in "good" episodes do not lead to immediate rewards, but instead transition towards states where better rewards are possible.

In retrospect, you do not know whether any single action was a good choice, but with the cross entropy method (CEM) you rely on the fact that on average the better episodes will contain more good decisions than the worse episodes, so you train the policy neural network on the state, action pairs from the elite episodes as if all the decisions were correct. This will not be true, but will hopefully be true more often than by chance, so the policy should improve.

This can be a noisy, high variance approach with any RL method. CEM is one of the most sensitive to noise and variance. However, the taxi environment is deterministic, and that makes using CEM more reasonable.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So, I should select all the state-action pairs in each episode with a total_reward bigger than the treshold. Am I ok? $\endgroup$
    – Hermes Morales
    Jul 22, 2021 at 11:59
  • 1
    $\begingroup$ @HermesMorales Yes. If an episode is "elite", then all the state/action pairs in it should be considered good for training data. $\endgroup$
    – Neil Slater
    Jul 22, 2021 at 12:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .