I am trying to create a fixed-topology MLP network from scratch (C#) which can classify some simple problems such as XOR and MNIST (Handwriting). The network will be trained purely with genetic algorithms instead of backprop.
Here are the details:
- Population size: 50
- Activation function: SIGMOID
- Fixed topology
- XOR: 2 inputs, 1 output. Tested w/ different numbers of hidden layers/nodes.
- MNIST: 28*28=784 inputs for all pixels, will be either ON(1) or OFF(0). 10 outputs to represent digits 0-9
- Initial population will be given random weights between 0 and 1
- 10 "Fittest" networks survive each iteration, and performs crossover to reproduce 40 offspring
- For all weights, mutation occurs to add a random value between -1 to 1, with a 5% chance
With 2 hidden layers of 4 and 3 neurons respectively, XOR managed to achieve 97-99.9% accuracy in around 100 generations. Biases were not used here.
However, trying out MNIST revealed a pretty glaring issue - the 784 inputs; a large increase of nodes compared to XOR, multiplied with weights and added up results in HUGE values of 50 to even 100, way beyond the typical domain range of the activation function (SIGMOID).
This just renders all layers' outputs as 1 or 0.99999-something, which breaks the entire network. Also, since this makes all individuals in a population extremely similar to one other, the genetic algorithm seems to have no clue on how to improve. Crossover will produce an offspring almost identical to its parents, and some lucky mutations are simply ignored my the sheer amount of other neurons!
What can be a viable solution to this? It's my first time studying NNs, and this is really... challenging. Please help!