I am making a school project where I should develop any kind of game where I can have one reactive agent and one agent based on machine learning competing with each other.
My game consists of a salesmen problem. Basically, I have 3 types of entities, consumers, salesmen, and hotspots.
The consumers are represented by the person with a green background. There are 8 of them. They basically move around the whole game using random walks and they tend o aggregate on the HotSpots (the orange icon with the router in it).
The salesmen are represented by the person with the dark grey background. One of them is controlled by a reactive agent that has in it some rules that I programmed and the other one is controlled by my DQN model.
The salesmen have 5 available actions, move up, right, down, left or sell. When they choose to sell the simulation will try to sell to the closest consumer in a predetermined max range. If no consumers exist in that range or if the consumer rejects to buy then the sell fails.
I started training a Deep Q Network that I built using TensorFlow. As input features, I am giving the agent current position, the position of each consumer and a boolean saying if the consumer was recently asked to buy or not (consumers that were asked to buy something will reject future offers for a determined amount of time with 100% probability). For the output layer, I have 5 nodes, one for each available action.
Here is a screenshot of the game: The red number on the right-bottom corner of each agent represents their total utility.
I decided to give the agents the following rewards:
- SELL_SUCCESSED_REWARD = 3 - The agents receives 3 points for each success sell.
- SELL_FAILED_REWARD = -0.010 - The agent loses 0.01 points for each failed sell
- MOVING_REWARD = -0.001 - The agent loses 0.001 points for each move
- NOT_MOVING_REWARD = -0.0125 - The agent loses 0.0125 points for standing in the same position (ie. not moving or trying to move against a wall)
I started training my agent but I seems to do not learn anything! I left it training for around 3 hours and I could not see any improvement. I tried different activation functions, batch sizes, exploration rates etc but no noticiable effect.
My question is: Can a DQN learn in this type of enviroment where there are a lot of random walks?
If yes what could be my problem? Not enought training time? Bad input features? Bad implementation?
Here are the files with my implementation of DQN: