After [checking the Internet][1], you will probably find several resources such as
* https://github.com/mohammadasghari/dqn-multi-agent-rl
* https://rlss.inria.fr/files/2019/07/RLSS_Multiagent.pdf
* https://arxiv.org/abs/2011.00583
and others.

Try to understand the principles first (see above). After some reasonable amount of coding you can adapt OpenAI gym.
Good luck!


  [1]: https://www.google.com/search?ei=2QtPYNbSItCIlwS3jIR4&q=github%20multi%20agent%20reinforcement%20learning&oq=github%20multi%20agent%20reinforcement%20learning