# Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max policies instead?

The main problem with using a softmax policy in Q learning* is that you have no independent set of preferences, just value estimates. So the agent's exploration performance would become dependent on the scaling of the reward signals. A simple method to adjust for that is to use a temperature hyperparameter $$T$$, that divides the action value estimates used in the softmax. A high temperature results in near random behaviour, and a low one will almost always select actions with the highest action value. You can start with a high temperature and slowly decay it, similar to decaying $$\epsilon$$ for $$\epsilon$$-greedy.