# Should the exploration rate be updated at the end of the episode or at every step?

My agent uses an $$\epsilon$$-greedy strategy to learn. The exploration rate (i.e. $$\epsilon$$) decays throughout the training. I've seen examples where people update $$\epsilon$$ every time an action is taken, while others update it at the end of the episode. If updated at every action, $$\epsilon$$ is more continuous. Does it matter? Is there a standard? Is one better than another?

• In the long run it would make no difference and both have their benefits. Updating after an episode will take longer to converge but offers more exploration and vice-versa. – David Ireland Oct 30 '20 at 21:51