So I have 2 models trained with the DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's and 1's (-1's for the other model)where 1 means that square is filled and 0 is empty. It is a map filling environment, where at each step the agent can move up, down, left or right and for each step, it stays alive without turning into itself (1 or -1 for the other) or the boundary of the environment it gets 0.005 rewards and for "dying" it gets -1. You can think of the player as in the game Tron, where it just leaves a trail behind. I stack the last 4 frames on top of each other so it knows which end is the "head". With a single agent, after training, I didn't get an optimal model which uses all the squares but it does manage to fill about 30% of the environment, which I think is the limit for this algorithm (let me know if you have thoughts on this)

Now, I put the two models in one environment where there are two players, one represented with 1's and the other with -1s. As one model is trained with -1's and the other with 1's I thought they could find their own player, however even before training if I just run the models on the environment without any exploration, they seem to affect the actions of each other. One just goes straight and dies and the other just turns once then dies at the wall (whereas in a single-agent environment these 2 models can fill about 30%). And if I do training, they just diverge to this exact behavior from random without seemingly not learning anything. So, I just wanted to ask is there anything wrong about my approach with the representation of the players (1 and -1) because I thought they would just play as they did in the single-agent environment but they don't and I couldn't get them to learn anything


When you trained your agents separately they never saw squares with opposing values (1/-1). So agents don't really know what to expect from visiting that square. I'd try adding (1/-1) to the condition on which you base your squares availability. Also try increasing the reward. It's hard to give suggestions without looking at the code.

  • $\begingroup$ I could add a wall of the opposite number that would kill them on single agent, would that adapt them to the other value? even if it's not a snake but a wall? I heard it's best to keep the rewards between -1 and 1, what would you suggest? $\endgroup$
    – Milky
    Jul 30 '19 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.