# How can I formalise a non-zero-sum game of $N$ agent as Markov game?

I coded a non-zero-sum game of $$N$$ agents in a discrete dynamic environment to RL with Q-learning and DQN agents.

It's like a marathon. Only two actions are available per agent: $$\{ G \text{ (move forward}) , S \text{ (stay to its position}) \}$$. Every agent has $$m$$ possible individual positions, the other agents cannot interfere with its path to the terminal position. Only when one agent reaches its terminal position gets a full reward. When everyone reaches a terminal state, all get $$0$$ rewards. If more than $$1$$ but less than $$N$$ reach their terminals, they get a small reward.

Now, I try to formalize it as a Markov Game (MG), but I don't have a solid mathematical background.

My first question is:

1. When we model a problem as an RL problem, the transition probability (TP) distribution is not required, while an MDP and MG require TP. But then how are all RL problems modeled first into MDP or MG?

As I have read in literature, I understand that I will treat the action sets of all other players as a "team" joint set of actions.

Second question:

1. How can I specialize the TP function to the specific problem I want to model? Should I just mention the general function equation?

What I have tried so far is to explicitly describe it, but I think I am not getting something:

1. The probability of the transition from $$s$$ to $$s'$$, where in $$s'$$ a number of $$k$$ agents move a step forward is equal to $$1$$, given that they all chose action $$G \in Α$$ and that the rest $$n-k$$, if any, all chose the action $$S \in Α$$, where $$k$$ is an integer $$1 \leq k \leq n$$.

2. The probability of the transition from $$s$$ to $$s'$$, where in $$s'$$ a number of $$k$$ agents to earn the high payoff is equal to $$1$$, given that $$k=1$$, its position is equal to $$m-1$$, it chooses action $$G \in A$$ and that the rest $$n-k$$ all chose the action $$S \in Α$$, where $$m$$ is the max possible position for each agent.

3. The probability of the transition from $$s$$ to $$s'$$, where in $$s'$$ a number of $$k$$ agents to earn a low payoff is equal to $$1$$, given that $$k>1$$, their position is equal $$m-1$$, they all chose action $$G \in Α$$ and that the rest $$n-k$$ all chose the action $$S \in Α$$, where $$k$$ is an integer with $$1 < k \leq n$$ and m is the max possible position for each agent

• Hi and welcome to AI SE! What definition of "Markov game" are you using? Can you link us to a paper that provides the definition? Furthermore, I suggest you ask either the 1st or the 2nd question in a separate. In general, try to ask only one question per post. – nbro Apr 14 '20 at 3:34