1
$\begingroup$

I have a neural network for MNIST classification which I am hard coding using TensorFlow 2.0. The neural network has an input layer consisting of 784 neurons (28 * 28), one hidden layer having "hidden_neurons" number of neurons and an output layer having 10 neurons.

The part of the code that I want to get checked is as follows:

# Partial derivative of cost function wrt b2-
dJ_db2 = (1 / m) * tf.reshape(tf.math.reduce_sum((A2 - Y), axis = 0), shape = (1, 10))

# Partial derivative of cost function wrt b1-
dJ_db1 = (1 / m) * tf.reshape(tf.math.reduce_sum(tf.transpose(tf.math.multiply(tf.matmul(W2, tf.transpose((A2 - Y))), relu_derivative(A1))), axis = 0), shape = (1, hidden_neurons))

The notation is as follows.

  • "b1" - bias for hidden layer and has the shape (1, hidden_neurons")
  • "b2" - bias for output layer having the shape (1, 10).
  • "A2" - is the output of output layer and have the shape (m, c)
  • "Y" - is one-hot encoded target and have the shape (m, c)
  • 'm' - is number of training examples
  • 'c' - is number of classes
  • "A1" - is the output of hidden layer and has the shape (hidden_neurons, m)

I have used multiclass cross-entropy cost function. Hidden layer uses ReLU activation function, while the output layer has softmax activation function.

Are my two lines of codes for cost function wrt to "b1" and "b2" correct?

$\endgroup$
  • $\begingroup$ Hi Arun and welcome to this community! It would be better if you could provide us the mathematical formulas that represent your two lines of code rather than the two lines of code themselves, but I think you want us to check that your implementation is correct. $\endgroup$ – nbro Nov 25 '19 at 1:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.