I am new to Reinforcement learning and am currently reading up on the estimation of Q $\pi(s, a)$ values using MC epsilon-soft approach and chanced upon this algorithm. The link to the algorithm is found from this website. https://www.analyticsvidhya.com/blog/2018/11/reinforcement-learning-introduction-monte-carlo-learning-openai-gym/ def monte_carlo_e_soft(env, episodes=100, policy=None, epsilon=0.01): if not policy: policy = create_random_policy(env) # Create an empty dictionary to store state action values Q = create_state_action_dictionary(env, policy) # Empty dictionary for storing rewards for each state-action pair returns = {} # 3. for _ in range(episodes): # Looping through episodes G = 0 # Store cumulative reward in G (initialized at 0) episode = run_game(env=env, policy=policy, display=False) # Store state, action and value respectively # for loop through reversed indices of episode array. # The logic behind it being reversed is that the eventual reward would be at the end. # So we have to go back from the last timestep to the first one propagating result from the future. # episodes = [[s1,a1,r1], [s2,a2,r2], ... [Sn, an, Rn]] for i in reversed(range(0, len(episode))): s_t, a_t, r_t = episode[i] state_action = (s_t, a_t) G += r_t # Increment total reward by reward on current timestep # if state - action pair not found in the preceeding episodes, # then this is the only time the state appears in this episode. if not state_action in [(x[0], x[1]) for x in episode[0:i]]: # # if returns dict contains a state action pair from prev episodes, # append the curr reward to this dict if returns.get(state_action): returns[state_action].append(G) else: # create new dictionary entry with reward returns[state_action] = [G] # returns is a dictionary that maps (s,a) : [G1,G2, ...] # Once reward is found for this state in current episode, # average the reward. Q[s_t][a_t] = sum(returns[state_action]) / len(returns[state_action]) # Average reward across episodes # Finding the action with maximum value. Q_list = list(map(lambda x: x[1], Q[s_t].items())) indices = [i for i, x in enumerate(Q_list) if x == max(Q_list)] max_Q = random.choice(indices) A_star = max_Q # 14. # Update action probability for s_t in policy for a in policy[s_t].items(): if a[0] == A_star: policy[s_t][a[0]] = 1 - epsilon + (epsilon / abs(sum(policy[s_t].values()))) else: policy[s_t][a[0]] = (epsilon / abs(sum(policy[s_t].values()))) return policy This algorithm computes the $Q(s, a)$ for all state action value pairs that the policy follows. If $\pi$ is a random policy, and after running through this algorithm, and for each state take the $\max Q(s,a)$ for all possible actions, why would that not be equal to $Q_{\pi^*}(s, a)$ (optimal Q function)? From this website, they claim to have been able to find the optimal policy when running through this algorithm. I have read up a bit on Q-learning and the update equation is different from MC epsilon-soft. However, I can't seem to understand clearly how these 2 approaches are different.