Is the temperature equal to epsilon in Reinforcement Learning?

Question

This is a piece of code from my homework.

# action policy: implements epsilon greedy and softmax
def select_action(self, state, epsilon):
    qval = self.qtable[state]
    prob = []
    if (self.softmax):
        # use Softmax distribution
        prob = sp.softmax(qval / epsilon)
        #print(prob)
    else:
        # assign equal value to all actions
        prob = np.ones(self.actions) * epsilon / (self.actions -1)
        # the best action is taken with probability 1 - epsilon
        prob[np.argmax(qval)] = 1 - epsilon
    return np.random.choice(range(0, self.actions), p = prob)

This is a method in order to select the best action according to the two polices i think. My question is, why in the softmax computation there is the epsilon parameter used as temperature. Is really the same thing? Are they different? I think they should be two different variables. Should the temperature be a fixed value over time? Because when i use the epsilon-greedy policy my epsilon decrease over time.

Answer 1

Your are correct that epsilon in epsilon-greedy and temperature parameter in the "softmax distribution" are different parameters, although they serve a similar purpose. The original author of the code has taken a small liberty with variable names in the select_action method in order to use just one simple name as a positional argument.

Should the temperature be a fixed value over time?

Not necessarily, if your goal is to converge on an optimal policy you will want to decrease temperature. A slow decay factor applied after each update or episode, as you might use for epsilon (e.g. 0.999 or other value close to 1), can also work for temperature decay. A very high temperature is roughly equivalent to epsilon of 1. As temperature becomes lower, differences in action value estimates become major differences in action selection probabilities, with a sometimes desirable effect of picking "promising" action choices more often, plus with very low probabilities of picking the actions with the worst estimates.

Using a more sophisticated action selection such as the temperature based on in the example code can speed learning in RL. However, this particular approach is only good in some cases - it is a bit fiddly to tune, and can simply not work at all.

The tricky part of using a temperature parameter is choosing good starting point, as well as the decay rate and end values (you have to do the last for epsilon-decay as well). The problem is that the impact of using this distribution depends on the actual differences between action choices. You need a temperature value on roughly the same scale as the Q value differences. This is difficult to figure out in advance. In addition, if the Q value differences are more pronounced in some states than in others, you risk either having next to no exploration or having too much in some parts of the problem.