I wrote two programs that simulated 10000 episodes in gym environment CartPole-v0.
The first program takes random moves in every steps in each episode. The average reward over 10000 episodes is 22.1582.
The second program uses a random policy in each episode. For each episode, initialize a 4 by 2 matrix $M$ with random numbers from a uniform distribution on $[0,1)$ that maps state observations to action values. Then choose the action with higher value in each step. The average reward over 10000 episodes is 46.8291.
The linear mapping given by $M$ covers only a portion of the search space so it seems the way actions are selected in the first program is more "random" than the second program. How can we go about explaining the huge discrepancy in the average rewards obtained by the two methods?