Generally, "perfect information" is not a formal trait of MDPs. There is a concept of the Markov property, but it only loosely coincides with "perfect information". For instance it is OK for there to be unknown/hidden state, provided it behaves effectively randomly (when revealed, it is drawn from a consistent distribution). An example might be a solo card game - or one where the opponent has a fixed policy such as most implementations of blackjack - which is fine as an MDP, and would not be a POMDP.
A "perfect information" game is chess. The current state of the board gives all the information required to play, and there is no additional state. When played against an opponent with a fixed policy, then this also has the Markov property. When played in general against many different opponent, and you cannot rely on the opponents' policies being the same as the training, then effectively it is non-Markov, or a POMDP. The plays for maximising reward depend on the opponent.
In many games though, there is the concept of "perfect play", of forcing the best possible result from the current game state. Planning/search routines using minimax approaches can theoretically solve these games, and reinforcement learning agents can learn them by assuming the opponent is also trying to play in this game-theoretic optimal approach. This is not quite the same as playing optimally against any given opponent - by comparison it is very safe and risk-averse compared to trying to trigger a mistake in the opponent due to something that the opponent may do. However, it often leads to stable solutions and strong players in RL.
Self play RL will tend to converge on the same game theoretic perfect play, provided it is stable in the given environment. An agent that uses the same learning engine for both players will progress through one player or the other making a mistake, and its opponent model will take advantage of that until the losing player finds a move that corrects its mistake.
To avoid this mistake/fix scenario becoming a stagnant loop of learning and unlearning, it is common to train the agent against multiple versions of itself (and sometimes other agents) to help it progress. Again this works provided the environment supports a stable game theoretic policy where it is possible to avoid mistakes against all adversarial choices. Most (possibly all? I am not sure of the theory here) two-player games are like this. When you extend to three or more players I believe this becomes a lot harder to figure out.
