What is a working configuration of a neuronal network (number of layers, lerning rate and so on) for a specific dataset?

Question

I try to solve some easy functions with a neuronal network (aforge-lib):

This is how I generate the dataset:

const int GesamtAnzahl = 200;
float[,] tempData = new float[GesamtAnzahl, 2];
float minX = float.MaxValue;
float maxX = float.MinValue;

Random rnd = new Random();
var granzen = new List<int>() 
{
    rnd.Next(1, GesamtAnzahl-1),
    rnd.Next(1, GesamtAnzahl-1),
    rnd.Next(1, GesamtAnzahl-1),
    rnd.Next(1, GesamtAnzahl-1),
};
granzen.Sort();

for (int i = 0; i < GesamtAnzahl; i++)
{

    var x = i;
    var y = -1;
    if ((i > granzen[0] && i < granzen[1]) ||
        (i > granzen[2] && i < granzen[3]))
    {
        y = 1;
    }
    tempData[i, 0] = x;
    tempData[i, 1] = y;
}

So this is quite easy: The output is 1 if the input is between the 2 lower random generated "borders" or between the 2 higher numbers. Otherwise the output is 1.

The input values are standardices to fit between -1 and 1. So 0 is -1 and 200 is 1.

As a network I used a BackPropagationLearning with a BipolarSigmoidFunction and several configurations like:

Learning Rate: 0,1
Momentum: 0
Sigmoids alpha value: 2
Hidden Layer 1: 4 neurons
Hidden Layer 2: 2 neurons


Learning Rate: 0,1
Momentum: 0
Sigmoids alpha value: 2
Hidden Layer 1: 4 neurons
Hidden Layer 2: 2 neurons
Hidden Layer 3: 2 neurons


Learning Rate: 0,2
Momentum: 0
Sigmoids alpha value: 2
Hidden Layer 1: 4 neurons
Hidden Layer 2: 2 neurons
Hidden Layer 3: 2 neurons

and so on. None of them worked. As described here: https://towardsdatascience.com/beginners-ask-how-many-hidden-layers-neurons-to-use-in-artificial-neural-networks-51466afa0d3e it should be enough to have 2 hidden layers. The first one with 4 neurons and the second one with 2.

The configurations which worked best were:

Learning Rate: 0,01
Momentum: 0
Sigmoids alpha value: 2
Hidden Layer 1: 4 neurons
Hidden Layer 2: 4 neurons
Hidden Layer 3: 4 neurons

Learning Rate: 0,02
Momentum: 0
Sigmoids alpha value: 2
Hidden Layer 1: 4 neurons
Hidden Layer 2: 2 neurons

This solves the problem about 50 % of the times.

As this is a quite simple problem I wonder if I am doing something wrong. I think there has to be a configuration which has better results.

What is the best configuration for this problem and why?

Additionally I tried:

• Having more data does not help. I created 5000 a dataset of 5000 points ( GesamtAnzahl = 5000). Then the networks have a even worse sucess rate.
• I tried to add an extra constant input (always 1) to the dataset but this also lowered the sucess rate

Answer 1

I ran a lot of randomly created networks to solve this problem, but none of the structures where able to reliably "solve" this problem.

Of course, some of them where able to solve it one time, some of them even twice but there where only one which solved it 3 times:

• LearningRate: 0,510141694690167
• Momentum: 0,962972165068133;
• Layer/Neuron-Count: 2 (14, 9)
• SigmundAlphaValue: 2;