I am trying to create a fixed-topology MLP from scratch (with C#), which can solve some simple problems, such as the XOR problem and MNIST classification. The network will be trained purely with genetic algorithms instead of back-propagation.
Here are the details:
- Population size: 50
- Activation function: sigmoid
- Fixed topology
- XOR: 2 inputs, 1 output. Tested with different numbers of hidden layers/nodes.
- MNIST: $28*28=784$ inputs for all pixels, will be either ON(1) or OFF(0). 10 outputs to represent digits 0-9
- Initial population will be given random weights between 0 and 1
- 10 "Fittest" networks survive each iteration, and performs crossover to reproduce 40 offspring
- For all weights, mutation occurs to add a random value between -1 to 1, with a 5% chance
With 2 hidden layers of 4 and 3 neurons respectively, XOR managed to achieve 97-99.9% accuracy in around 100 generations. Biases were not used here.
However, trying out MNIST revealed a pretty glaring issue - the 784 inputs; a large increase of nodes compared to XOR, multiplied with weights and added up results in HUGE values of 50 to even 100, way beyond the typical domain range of the activation function (sigmoid).
This just renders all layers' outputs as 1 or 0.99999-something, which breaks the entire network. Also, since this makes all individuals in a population extremely similar to one other, the genetic algorithm seems to have no clue on how to improve. The crossover will produce an offspring almost identical to its parents, and some lucky mutations are simply ignored by the sheer amount of other neurons!
What can be a viable solution to this?
It's my first time studying NNs, and this is really challenging.