# Summed weights are too big for activation function [GANN]

I am trying to create a fixed-topology MLP network from scratch (C#) which can classify some simple problems such as XOR and MNIST (Handwriting). The network will be trained purely with genetic algorithms instead of backprop.

Here are the details:

• Population size: 50
• Activation function: SIGMOID
• Fixed topology
• XOR: 2 inputs, 1 output. Tested w/ different numbers of hidden layers/nodes.
• MNIST: 28*28=784 inputs for all pixels, will be either ON(1) or OFF(0). 10 outputs to represent digits 0-9
• Initial population will be given random weights between 0 and 1
• 10 "Fittest" networks survive each iteration, and performs crossover to reproduce 40 offspring
• For all weights, mutation occurs to add a random value between -1 to 1, with a 5% chance

With 2 hidden layers of 4 and 3 neurons respectively, XOR managed to achieve 97-99.9% accuracy in around 100 generations. Biases were not used here.

However, trying out MNIST revealed a pretty glaring issue - the 784 inputs; a large increase of nodes compared to XOR, multiplied with weights and added up results in HUGE values of 50 to even 100, way beyond the typical domain range of the activation function (SIGMOID).

This just renders all layers' outputs as 1 or 0.99999-something, which breaks the entire network. Also, since this makes all individuals in a population extremely similar to one other, the genetic algorithm seems to have no clue on how to improve. Crossover will produce an offspring almost identical to its parents, and some lucky mutations are simply ignored my the sheer amount of other neurons!

What can be a viable solution to this? It's my first time studying NNs, and this is really... challenging. Please help!

In addition, once trained, they tend to follow a certain size distribution. It helps if you start with values within that range. This is often called Xavier or Glorot initialisation. Basically, if your number of inputs to a layer is n_inputs and number of outputs is n_outputs, you will have n_inputs * n_outputs weights, and if you initialise them with a random number generator, then you should use a multiplier something like this:
w = (rand() - 0.5) * sqrt(6/(n_inputs + n_outputs))

As you are using a GA, you may want to use some form of clipping or normalisation to prevent mutations from drifting too far away from these useful weights. I'm not entirely sure what would be best, as usually I would use back propagation to train a large network. Potentially a maxnorm regularisation would help - choose a value (maybe even make it adjustable via mutation) that the norm of weight matrix in each layer should not exceed (separately for each layer, or one overall value, up to you), and if any learning step creates a network with too high a norm, scale it down. The L2 norm, sqrt(sum_of_squared_weights) is usual choice for maxnorm regularisation, and this works well with backpropagation-based supervised learning.