1
$\begingroup$

So I'm trying to make my first neural network and have just finished my back propagation functions. I got the algebra from brilliant and thought I'd understood it, but my bug proves otherwise. The bug itself is that after 500 iterations, both of my output neurons give a value of 1 despite my target values being 0 or 1. None of my training data gives both neurons outputting 1 at the same time, yet it happens. I'm sorry for posting a lot of code here, but I can't think of a better way of showing off the potential problem areas.

This calculates the error term for each neuron:

// Calculates error for each neuron in this layer, which we know isn't the output layer
// Error Term = derivation of sigmoid * sum of (weight to next layer * error term in next layer)
prevLayer.neurons.at(neuron).neuronData.error = prevLayer.neurons.at(neuron).CalculateSumWeightsError(this->neurons) *
    prevLayer.neurons.at(neuron).GetDerivedValue(theta);

This is the contents of the loop that calculates the delta weights for each connection:

            // Calculates the connections new weight
            // delta weight = 
            // -learning rate * 
            // Derivative of activation * 
            // input * 
            // sum of (weights * error term of next layer)

        float newWeight = alpha * prevLayer.neurons.at(neuron).GetDerivedValue(theta);
        newWeight *= prevLayer.neurons.at(neuron).neuronData.value;
        newWeight *= this->neurons.at(errorNeuron).neuronData.sumErrorWeights;

This finds the total error and individual error terms in the output layer. It's part of a loop that iterates through all of them.

// Calculate individual error terms
// Error Terms = target - output
neuralLayers.back().neurons.at(output).neuronData.error = targets[output] - neuralLayers.back().neurons.at(output).GetActivatedValue(theta);
// Calculate Total Error for the output layer
// Total Error = sum of 1/2*(target - output)^2
errorTotal += 0.5f * FMath::Pow((targets[output] - neuralLayers.back().neurons.at(output).GetActivatedValue(theta)), 2.f);

Like I said, this is my first attempt at a neural network and thought I'd got it right this time round. Any help or resources would be very much appreciated as I genuinely don't know what's gone wrong.

$\endgroup$
1
  • $\begingroup$ I don't have time to comb over the code, but when I was having gradient issues with my CNN, I found that doing a manual (ie, pen and paper) forward and backward pass of a simple architecture helped clean up any calculation issues quickly. You should try doing the same, take values at all layers and ensure they match what you manually calculated. $\endgroup$
    – Recessive
    Jan 12 at 4:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.