I am trying to ascertain if my neural network is able to generalize or if it’s simply using memory/overfitting to solve a task. I would like my model to generalise.
Currently, I train the neural network on a randomly generated 3x3 frozen lake environment - with no holes. (The network simply chooses an action for each state it is presented.)
Then, I test the model on a much larger frozen lake environment. Still no holes. Still randomly generated. The test environment size is assigned by a random value of 5-15 for each axis (height/width), randomly generated.
Then I determine the "degree of generalization" by how many large environments the network is able to solve. At present, it solves 100/100 on the 3x3, and about 83/100 on the larger test environments.
When I track the solutions it generates, I can see that the network always takes the shortest route available, which is great.
Do you guys have any ideas, inputs or criticism on the method I use to determine the degree of generalization?