I'm using Unity Ml-agents to learn more about reinforcement learning. I've created the most basic environment I could think of, but I'm getting some interesting results.
Take a look at the picture below, these are the results of an agent training for 500k steps, all with the same hyperparameters and the same run-seed. The only thing that I changed was the preprocessing of the inputs. More details below the results;
The environment is as follows; the agent lives on a 10x10 grid where a target is placed on a random location. The agent spawns in the center of the grid, position 0,0,0. The agent simply gets a reward of 1 whenever it reaches the goal, and the episode ends whenever the agent falls of the grid, or the maximum allowed stepcount is reached.
The agent receives its own velocity in the x and z-axis, the location of itself, and the location of the target.
The agent's output is a continuous action space of 2 values. One for left and right movement and one for up and down. Both values between -1 and 1.
So here's what I did to the inputs;
- Orange: just the raw inputs, nothing changed. Result: Baseline performance
- Dark blue: normalised between 0 and 1. Result: Really bad performance, not converging
- Dark red: normalised between 0 and 10. Result: converging, but much slower
- Light blue: normalized between -1 and 1. Result: better performance that 0 to 10, but worse than baseline
- Pink: normalised between -10 and 10. Result: on par with baseline performance
Can anyone help me understand what's going on here? I was under the impression that normalising inputs would result in better learning, but these results tell a whole different story.