How to represent players in a multi agent environment so each model can distinguish its own player

Question

So I have 2 models trained with the DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's and 1's (-1's for the other model)where 1 means that square is filled and 0 is empty. It is a map filling environment, where at each step the agent can move up, down, left or right and for each step, it stays alive without turning into itself (1 or -1 for the other) or the boundary of the environment it gets 0.005 rewards and for "dying" it gets -1. You can think of the player as in the game Tron, where it just leaves a trail behind. I stack the last 4 frames on top of each other so it knows which end is the "head". With a single agent, after training, I didn't get an optimal model which uses all the squares but it does manage to fill about 30% of the environment, which I think is the limit for this algorithm (let me know if you have thoughts on this)

Now, I put the two models in one environment where there are two players, one represented with 1's and the other with -1s. As one model is trained with -1's and the other with 1's I thought they could find their own player, however even before training if I just run the models on the environment without any exploration, they seem to affect the actions of each other. One just goes straight and dies and the other just turns once then dies at the wall (whereas in a single-agent environment these 2 models can fill about 30%). And if I do training, they just diverge to this exact behavior from random without seemingly not learning anything. So, I just wanted to ask is there anything wrong about my approach with the representation of the players (1 and -1) because I thought they would just play as they did in the single-agent environment but they don't and I couldn't get them to learn anything

Answer 1

When you trained your agents separately they never saw squares with opposing values (1/-1). So agents don't really know what to expect from visiting that square. I'd try adding (1/-1) to the condition on which you base your squares availability. Also try increasing the reward. It's hard to give suggestions without looking at the code.

Answer 2

You seem to use 0.0 for "free", which is exactly in the middle between the two "danger" values -1.0 and +1.0. This makes it impossible to separate danger vs free with just a linear mapping.

So every of your inputs requires learning to solve a XOR problem first, it seems. This means multiple layers are a must to make progress. This should be learnable given enough time, but... did you intentionally make the problem so very hard?