I wrote two programs that simulated 10000 episodes in gym environment CartPole-v0.

The first program takes random moves in every steps in each episode. The average reward over 10000 episodes is 22.1582.

The second program uses a random policy in each episode. For each episode, initialize a 4 by 2 matrix $M$ with random numbers from a uniform distribution on $[0,1)$ that maps state observations to action values. Then choose the action with higher value in each step. The average reward over 10000 episodes is 46.8291.

The linear mapping given by $M$ covers only a portion of the search space so it seems the way actions are selected in the first program is more "random" than the second program. How can we go about explaining the huge discrepancy in the average rewards obtained by the two methods?

  • $\begingroup$ Cartpole has far more than 4 by 2 state/actions. How are you mapping from state to lookup of action value in your 4 by 2 table? I would guess that the 2 corresponds to "move left, move right" evaluation. But how are you altering the state in order to look it up in the table? $\endgroup$ – Neil Slater Jan 16 at 9:30
  • $\begingroup$ Matrix $M$ is not a lookup table. It is a matrix that maps a vector of size 4 (representing the state) to a vector of size 2 (representing the action values) $\endgroup$ – artils1997 Jan 17 at 2:52
  • $\begingroup$ So you are using the matrix to multiply the state vector? Essentially that is a linear value approximation. Except the values will be meaningless/wrong, so essentially it's a randomised policy with a few inflection points between left/right actions $\endgroup$ – Neil Slater Jan 17 at 10:14
  • $\begingroup$ Yes they are randomly generated policies. Basically, I am just wondering why randomly generated policies, on average, perform better than taking random actions in terms of the average returns from each episode. $\endgroup$ – artils1997 Jan 17 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.