So I'm trying to make my first neural network and have just finished my back propagation functions. I got the algebra from brilliant and thought I'd understood it, but my bug proves otherwise. The bug itself is that after 500 iterations, both of my output neurons give a value of 1 despite my target values being 0 or 1. None of my training data gives both neurons outputting 1 at the same time, yet it happens. I'm sorry for posting a lot of code here, but I can't think of a better way of showing off the potential problem areas.
This calculates the error term for each neuron:
// Calculates error for each neuron in this layer, which we know isn't the output layer // Error Term = derivation of sigmoid * sum of (weight to next layer * error term in next layer) prevLayer.neurons.at(neuron).neuronData.error = prevLayer.neurons.at(neuron).CalculateSumWeightsError(this->neurons) * prevLayer.neurons.at(neuron).GetDerivedValue(theta);
This is the contents of the loop that calculates the delta weights for each connection:
// Calculates the connections new weight // delta weight = // -learning rate * // Derivative of activation * // input * // sum of (weights * error term of next layer) float newWeight = alpha * prevLayer.neurons.at(neuron).GetDerivedValue(theta); newWeight *= prevLayer.neurons.at(neuron).neuronData.value; newWeight *= this->neurons.at(errorNeuron).neuronData.sumErrorWeights;
This finds the total error and individual error terms in the output layer. It's part of a loop that iterates through all of them.
// Calculate individual error terms // Error Terms = target - output neuralLayers.back().neurons.at(output).neuronData.error = targets[output] - neuralLayers.back().neurons.at(output).GetActivatedValue(theta); // Calculate Total Error for the output layer // Total Error = sum of 1/2*(target - output)^2 errorTotal += 0.5f * FMath::Pow((targets[output] - neuralLayers.back().neurons.at(output).GetActivatedValue(theta)), 2.f);
Like I said, this is my first attempt at a neural network and thought I'd got it right this time round. Any help or resources would be very much appreciated as I genuinely don't know what's gone wrong.