I'm working on implementation of the backpropagation algorithm for a simple neural network which predicts a probability of survival (1 or 0) and I can't get it above 80% no matter how much I try to set the right hyperparameters. I suspect that's because my backpropagation is implemented incorrectly since I tried 2 different types of code and both give me same results. Is my backpropagation implemented correctly? Also how can I improve my model to give a better prediction?

class NeuralNetwork(object):

def __init__(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
    # Set number of nodes in input, hidden and output layers.
    self.input_nodes = input_nodes
    self.hidden_nodes = hidden_nodes
    self.output_nodes = output_nodes
    self.lr = learning_rate

    # Initialize weights
    self.input_hidden_weights = np.random.randn(hidden_nodes, input_nodes) # 10x7
    self.hidden_output_weights = np.random.randn(output_nodes, hidden_nodes) # 1x10

    # Sigmoid activation funciton
    self.sigmoid = lambda x: 1/(1+np.exp(-x))
    self.diff_sigm = lambda x: x*(1-x)

def train(self, input_list, label_list):

    # Create an array of inputs and labels
    inputs = np.array(input_list, ndmin=2).T # 7x1
    labels = np.array(label_list, ndmin=2) # 1x1

    # Forward propagation
    hidden_layer = self.sigmoid(np.dot(self.input_hidden_weights, inputs))        
    output_layer = self.sigmoid(np.dot(self.hidden_output_weights, hidden_layer))

    final_output = output_layer

    # Error function
    output_errors = labels-final_output

    # Backpropagation  
    output_delta = output_errors * self.diff_sigm(output_layer)
    hidden_delta = np.dot(self.hidden_output_weights.T, output_delta) * self.diff_sigm(hidden_layer)

    # Update the weights
    self.hidden_output_weights += np.dot(output_delta, hidden_layer.T) * self.lr
    self.input_hidden_weights += np.dot(hidden_delta, inputs.T) * self.lr

    # Backpropagation
    hidden_errors = np.dot(self.hidden_output_weights.T, output_errors)        
    hidden_grad = hidden_layer * (1.0 - hidden_layer)

    # Update the weights
    self.hidden_output_weights += self.lr * np.dot(output_errors.T, output_layer.T) # update hidden-to-output weights with gradient descent step    
    self.input_hidden_weights += self.lr * np.dot(hidden_errors * hidden_grad, inputs.T)  # update input-to-hidden weights with gradient descent step
  • $\begingroup$ Welcome to AI! (I suspect this particular question would be more appropriate for Cross Validated or Data Science. I'll hold off on taking action pending other opinion, but let me know if you want me to migrate this to one of the linked stacks.) $\endgroup$
    – DukeZhou
    Sep 3 '17 at 21:52
  • $\begingroup$ Sorry I didn't know. Yes, that'd be very helpful, thanks. $\endgroup$ Sep 4 '17 at 1:55

For future reference, I will merely point you to a technique you can implement to test the correctness or lack thereof, of your backpropagation implementation.

Ps: don't feel too bad for having gotten it slightly wrong, "backpropagation is notoriously difficult to implement" - source :).

In fact, there is a technique called "Gradient checking" meant specifically for this purpose (for more information, see Andrew Ng's lecture video on gradient checking and, this notes). I would argue that even gradient checking is a little tricky to implement.

How does Gradient checking work:

  • Backpropagation computes the gradients (∂J/∂θ)(∂J/∂θ), where θ denotes the parameters of the model. J is computed using forward propagation and your loss function.
  • But because forward propagation is fairly straightforwards to implement, most people are usually confident that you got its implementation correct. So the trick is to use the value of J to verify your code for computing (∂J/∂θ)(∂J/∂θ).
  • We know that by definition, the gradient or derivative is given by:

    (∂J/∂θ) = lim ε→0 J(θ + ε) − J(θ − ε) / 2ε 

now, since we trust our calculation of (∂J/∂θ), we can easily compute the value of J(θ+ε)J(θ+ε) and J(θ−ε)J(θ−ε).

For more information, see here and, over here).


For future reference, you can check your correctness by the finite difference method.

http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.3-BackProp.pdf (p.23)


The given so far answers focus on numerical methods to check your gradients. It is really useful, especially if one doesn't have much experience in backprop.

But I'd like to add here a pure practical "sanity check", relatively fast and easy to perform, which also works for other issues, e.g. (rough) hyperparameters selection. To see if your network makes sense, reduce the training set to a few examples and try to overfit the network. If the loss falls to zero and training accuracy skyrockets to 1, it means that both passes work correctly and can move on to the real training. Otherwise, something's not right and should dive into specific parts of the network, in particular check the gradients numerically.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.