You should start with the general definition of Reinforcement Learning problem. And what Markov Decision Process is.
DQN, A3C, PPO and REINFORCE are algorithms for solving reinforcement learning problems. These algorithms have their strengths and weaknesses depending on the details of the underlying problem.
Multi-Armed Bandit is not even an algorithm - it is a subclass of reinforcement learning problems, where your environment (usually) doesn't have any state transitions and your actions are just a single choice from (usually) fixed and finite set of choices.
Multi-Armed Bandit is used as an introductory problem to reinforcement learning, because it illustrates some basic concepts in the field: exploration-exploitation tradeoff, policy, target an estimate, learning rate and gradient optimization. All these concepts are basic vocabulary in RL. I recommend reading (and, very importantly, doing all the exercises) the Sutton and Barto book chapter two to get familiarized with it.
Edit: since the answer got popular, I'll address the comments and the question edit.
Being a special simplified subset of Markov Decision Processes, Multi Armed Bandit problems allow deeper theoretical understanding. For example, (as per @NeilSlater comment) the optimal policy would be to always go for the best arm. So it makes sense to introduce "regret" $\rho$ - the difference between a potential optimal reward and the actual collected reward by agent following your strategy:
$$\rho(T) = \mathbb{E}\left[T\mu^* -\sum_{t=1}^T\mu(a_t)\right]$$
One can then study asymptotic behavior of this regret as a function of $T$ and devise strategies with different asymptotic properties. As you can see, the reward here is not discounted ($\gamma=1$) - we usually can study the behavior of it as a function of $T$ without this regularization.
Although, there is one famous result that uses discounted rewards - the Gittins index policy (note, though, that they use $\beta$ instead of $\gamma$ to denote the factor).
