i have trouble implementing back propogation for multi class classification of CIFAR10 dataset

My neural network has 2 layers

forward propagation

X -> L1 -> L2

weights W are initialized as random

np.random.randn(this_layer_units, previous_layer_units) * 0.01

X is input of size (no_features * number of examples)

Z1 = (w1 * x) + b1

A1 = relu(Z1)

L1 has ReLu activation

Z2 = (w2 * A1) + b2

A2 = softmax(Z1)

L2 has softmax activation

cost is caluclated using this equation

cost = -(1/m)*np.sum((Y * np.log(A2) ) + ((1 - Y)*np.log(1-A2)))

back propagation

derivative of cost is calculated

dA2 = -(1/m)*(np.divide(Y, A2) - np.divide(1 - Y, 1 - A2))

dA2 = derivative of A2

Y = one hot encoded True values

softmax is

np.exp(z)/ np.sum(np.exp(z))

now how do i proceed from here

how do i find dZ2 (derivative of Z2) using dA2

and update weights

Link to entire jupyter notebook code


softmax is a Classifier with cross entropy loss function.

good explanation on softmax at below link http://cs231n.github.io/linear-classify/

code to implement softmax with X->RELU->Softmax with back prop http://cs231n.github.io/neural-networks-case-study/#together

After RELU, scores need to be exponentiate and calculate class probabilities and compute average loss for correct log probabilities and compute gradient on scores and then apply back prop - code for back prop is in above link.

  • $\begingroup$ Please try to provide comprehensive answers instead of link only answers...Since the user is a beginner it is assumed the OP has gone through other blogs to implement the classifier...so linking to another article is not of much help. $\endgroup$ – DuttaA Sep 21 '18 at 18:23
  • $\begingroup$ the link has full code on how to implement back prop with softmax $\endgroup$ – Sreedhar Veluri Sep 21 '18 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.