After checking the Internet, you will probably find several resources such as
- https://github.com/mohammadasghari/dqn-multi-agent-rl
- https://rlss.inria.fr/files/2019/07/RLSS_Multiagent.pdf
- https://arxiv.org/abs/2011.00583 and others.
Try to understand the principles first (see above). After some reasonable amount of coding you can adapt OpenAI gym. Good luck!
Update 17 March 2022:
You may want to check this popular repository as well https://github.com/Farama-Foundation/PettingZoo
Update 06 August 2023:
The best multi-agent tutorial I have seen so far comes from RLlib documentation. See RLlib for muti-agent RL.