There is an additional factor to consider about exploration/exploitation trade-off, that sometimes applies in addition to the reason in the accepted answer and most other answers here.
Sometimes an agent is required to both act and train itself in a "real" system, or at least one where the rewards are more than just collected training data from a simulation, but also represent actual profits and losses realised by the agent.
This is a common feature of adaptive content display in advertising, which typically uses the simpler k-armed bandit or contextual bandit model - still related to RL, and importantly still affected by the exploration/exploitation trade-off. It is very hard for a machine to model how humans will respond to an advert, so the only reliable measurements are made in production. Each click-through is then real money to someone, so it is important to adapt quickly to incoming data - but due to the variance in results it is also important to still keep testing the non-optimal choice and improve on any early estimates.
In such a scenario, you have to accept some non-optimal returns as the cost of finding the best ones through trial and error. However, it is important to balance this with gaining as much reward as possible whilst training.
So it can be a more important consideration, to obtain best cumulative reward during an ongoing training process, than even finding the optimal policy. That means taking care to balance exploitation and exploration, often strongly favouring exploitation after a relatively short high exploration phase.