0
$\begingroup$

I am writing a Neural Network frorm scratch. Below is what I have right now, based off of the math that I think I understand.

##### Imports #####
from matplotlib import pyplot as plt
import numpy as np

###### Activation Function #####
def sigmoid(input, derivative = False):
    if derivative:
        return sigmoid(input) * (1 - sigmoid(input))
    return 1 / (1 + np.exp(-input))

##### Feed Forward Neural Netowkr Class #####
class FFNN:
    def __init__(self, learning_rate, num_epochs):
        # Network
        self.w1 = np.random.randn(30, 5)
        self.w2 = np.random.randn(5, 3)

        # Hyperparameters
        self.learning_rate = learning_rate
        self.num_epochs = num_epochs

    # Forward Propagation
    def forward(self, input):
        self.z1 = np.dot(input, self.w1)
        self.a1 = sigmoid(self.z1)

        self.z2 = np.dot(self.a1, self.w2)
        self.a2 = sigmoid(self.z2)
    
    # Backward Propagation
    def backward(self, input, error):
        error2 = error * sigmoid(self.z2, derivative = True)
        d2 = np.dot(self.a1.T, error2)

        error1 = np.dot(self.w2, error2.T).T * sigmoid(self.z1, derivative = True)
        d1 = np.dot(input.T, error1)

        self.w1 -= d1 * self.learning_rate
        self.w2 -= d2 * self.learning_rate

    # Train
    def train(self, inputs, labels):
        for _ in range(self.num_epochs):
            for input, label in zip(inputs, labels):
                self.forward(input)
                self.backward(input, self.a2 - label)

    # Test
    def test(self, inputs):
        for input in inputs:
            self.forward(input)

            print('Image is a', 'ABC'[np.argmax(self.a2)])
            plt.imshow(input.reshape(5, 6))
            plt.show()

# Initialize Neural Network
feed_forward_neural_network = FFNN(learning_rate = 0.1, num_epochs = 100)

##### Training #####
a = [0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1]
b = [0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0]
c = [0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0]

y = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]])
x = [np.array(a).reshape(1, 30), np.array(b).reshape(1, 30), np.array(c).reshape(1, 30)]

feed_forward_neural_network.train(x, y)

##### Testing #####
feed_forward_neural_network.test(x)

However, after looking at someone else's code, they have the same thing except the backward function does this instead:

    # Backward Propagation
    def backward(self, input, error):
        error2 = error
        d2 = np.dot(self.a1.T, error2)

        error1 = np.dot(self.w2, error2.T).T * sigmoid(self.z1, derivative = True)
        d1 = np.dot(input.T, error1)

        self.w1 -= d1 * self.learning_rate
        self.w2 -= d2 * self.learning_rate

Notice the missing sigmoid(self.z2, derivative = True) multiplication by the layer 2 error. Both of these functions converge just fine, but obviously one of them is wrong. Which one, and why?

$\endgroup$

2 Answers 2

1
$\begingroup$

Your $d_2$ is the gradient used to update $w_2$, which is of course $\frac{dL}{dw_2}$. To compute this gradient, using your notation: $$ \frac{dL}{dw_2} = \frac{dL}{da_2}\frac{da_2}{dz_2}\frac{dz_2}{dw_2} = err \cdot \sigma'(z_2)\cdot a_1$$ So your version seems to be correct.

One possibility is that the forward is also different, and there is no sigmoid after the second layer in your colleague's network (which is often the case for the last layer). In which case their version would also be correct.

$\endgroup$
2
  • $\begingroup$ I check their forward, as that was what I thought originally, but they're the same. Good to hear I haven't misunderstood though. Thanks. $\endgroup$
    – user55176
    Jul 5, 2022 at 19:08
  • $\begingroup$ Please don't forget to accept and/or upvote answers. Thanks. $\endgroup$
    – Martino
    Jul 6, 2022 at 8:06
0
$\begingroup$

The latter one seems to correct implementation assuming your loss function is binary cross-entropy. The partial derivative for cross-entropy w.r.t z2 is self.a2 - label in your example. You can check mathematics in more detail here.

So, there is no need to multiply it again with derivative of sigmoid.

$\endgroup$
2
  • $\begingroup$ I don't think I'm using Binary Cross Entropy Loss? The error I backpropagated was just the output of the network - the real label. Cross Entropy involves logarithms does it not? $\endgroup$
    – user55176
    Jul 5, 2022 at 19:07
  • $\begingroup$ May I know, what is the loss function that you are using here and if you are using least squares then your version seems to be fine. Though people generally use binary-crossentropy for binary classification (and sigmoid activation for last layer). Finally, both implementation be right based on the loss function that you are using. $\endgroup$ Jul 6, 2022 at 7:05

You must log in to answer this question.