Even though if exploration doesn't happen, it's deterministic.
I would argue it is just stochastic because it chooses the current best action with probability $1-\epsilon+\epsilon/|A|$ and then selects randomly among the rest of the actions with the remaining probability $\epsilon/|A|$, where $A$ is the action space. The current best action is updated over time with running averages and may be the same one in the long run if it is truly a stationary bandit environment, but it will still explore.
Yes - you can think of an epsilon-greedy policy as a mixture of a policy that chooses an action at random (the stochastic part) and a possibly deterministic policy used otherwise. The value of epsilon gives the weight of the random component, and $1-\epsilon$ that of the other component.
When a policy is stochastic, it means taking actions will be done based on probabilities. For example, in deterministic policy (in the case of navigation for four actions), if an agent takes upward action, it will go up and etc. However, in stochastic policy, if an agent takes upward, it may go up with 80% of probability and go right with 20% of possibility. Thus, in epsilon-greedy based policies, we choose actions in exploration mode randomly, but it does not mean it is stochastic. I tried to explain short and clear.