What are the most common non-Markov RL paradigms?

Question

I am interested in doing model-free RL but not using the Markov assumptions typical for MDPs or POMDPs.

What are alternative paradigms that don't rely on the Markov assumptions? Are there any common approaches when this assumption is violated?

EDIT: I am asking for mathematical models that do not make the Markov assumption and so could be used for problems where the Markov assumption does not hold