# How to solve the problem of too big activations when using genetic algorithms to train neural networks?

I am trying to create a fixed-topology MLP from scratch (with C#), which can solve some simple problems, such as the XOR problem and MNIST classification. The network will be trained purely with genetic algorithms instead of back-propagation.

Here are the details:

• Population size: 50
• Activation function: sigmoid
• Fixed topology
• XOR: 2 inputs, 1 output. Tested with different numbers of hidden layers/nodes.
• MNIST: $$28*28=784$$ inputs for all pixels, will be either ON(1) or OFF(0). 10 outputs to represent digits 0-9
• Initial population will be given random weights between 0 and 1
• 10 "Fittest" networks survive each iteration, and performs crossover to reproduce 40 offspring
• For all weights, mutation occurs to add a random value between -1 to 1, with a 5% chance

With 2 hidden layers of 4 and 3 neurons respectively, XOR managed to achieve 97-99.9% accuracy in around 100 generations. Biases were not used here.

However, trying out MNIST revealed a pretty glaring issue - the 784 inputs; a large increase of nodes compared to XOR, multiplied with weights and added up results in HUGE values of 50 to even 100, way beyond the typical domain range of the activation function (sigmoid).

This just renders all layers' outputs as 1 or 0.99999-something, which breaks the entire network. Also, since this makes all individuals in a population extremely similar to one other, the genetic algorithm seems to have no clue on how to improve. The crossover will produce an offspring almost identical to its parents, and some lucky mutations are simply ignored by the sheer amount of other neurons!

What can be a viable solution to this?

It's my first time studying NNs, and this is really challenging.

In addition, once trained, they tend to follow a certain size distribution. It helps if you start with values within that range. This is often called Xavier or Glorot initialisation. Basically, if your number of inputs to a layer is n_inputs and number of outputs is n_outputs, you will have n_inputs * n_outputs weights, and if you initialise them with a random number generator, then you should use a multiplier something like this:
w = (rand() - 0.5) * sqrt(6/(n_inputs + n_outputs))

As you are using a GA, you may want to use some form of clipping or normalisation to prevent mutations from drifting too far away from these useful weights. I'm not entirely sure what would be best, as usually I would use back propagation to train a large network. Potentially a maxnorm regularisation would help - choose a value (maybe even make it adjustable via mutation) that the norm of weight matrix in each layer should not exceed (separately for each layer, or one overall value, up to you), and if any learning step creates a network with too high a norm, scale it down. The L2 norm, sqrt(sum_of_squared_weights) is usual choice for maxnorm regularisation, and this works well with backpropagation-based supervised learning.