# Stochastic gradient descent does not behave as expected, even with different activation functions

I have been working on my own AI for a while now, trying to implemented SGD with momentum from scratch in python. After looking around and studying all the maths behind it, i finally managed to implement SGD in a neural network that i trained to recognize the classic MNIST digits dataset. As activation function i always used sigmoid for both hidden and output neurons, and everything seems to work more or less ok, but now i wanted to step it up a bit and try to let SGD operate with different activations, so i added 2 other functions to my code: relu and tanh. The behaviours that i expected based on articles, documentation and "tutorials" found online were:
tanh: Should be slightly better than sigmoid
relu: should be much better than sigmoid and tanh
(By better i mean faster or at least higher accuracy the the end, or a mix of both)

Using tanh it looks like it's much slower converging to a minimum compared to sigmoid

Using relu...well, the results were very, VERY horrible Here's the outputs with the different activations (Learning rate: 0.1, Epochs: 5, MiniBatch size: 10, Momentum: 0.9)

Sigmoid training

[Sigmoid for hidden layers, sigmoid for output layer]
Epoch: 1/5 (14.3271 s): Loss: 0.0685, Accuracy: 0.6231, Learning rate: 0.10000
Epoch: 2/5 (14.0060 s): Loss: 0.0503, Accuracy: 0.6281, Learning rate: 0.10000
Epoch: 3/5 (14.0081 s): Loss: 0.0482, Accuracy: 0.6382, Learning rate: 0.10000
Epoch: 4/5 (13.8516 s): Loss: 0.0471, Accuracy: 0.7085, Learning rate: 0.10000
Epoch: 5/5 (13.9411 s): Loss: 0.0374, Accuracy: 0.7990, Learning rate: 0.10000


Tanh training

[Tanh for hidden layers, sigmoid for output layer]
Epoch: 1/5 (13.7553 s): Loss: 0.3708, Accuracy: 0.4171, Learning rate: 0.10000
Epoch: 2/5 (13.7666 s): Loss: 0.2580, Accuracy: 0.4623, Learning rate: 0.10000
Epoch: 3/5 (13.5550 s): Loss: 0.2289, Accuracy: 0.4824, Learning rate: 0.10000
Epoch: 4/5 (13.7311 s): Loss: 0.2211, Accuracy: 0.5729, Learning rate: 0.10000
Epoch: 5/5 (13.6996 s): Loss: 0.2142, Accuracy: 0.5779, Learning rate: 0.10000


Relu training

[Relu for hidden layers, sigmoid for output layer]
Epoch: 1/5 (14.2100 s): Loss: 0.7725, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 2/5 (14.6218 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 3/5 (14.2116 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 4/5 (14.1657 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 5/5 (14.1427 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000


Another run with relu

Epoch: 1/5 (14.7391 s): Loss: 15.4055, Accuracy: 0.1658, Learning rate: 0.10000
Epoch: 2/5 (14.8203 s): Loss: 59.2707, Accuracy: 0.1709, Learning rate: 0.10000
Epoch: 3/5 (15.3785 s): Loss: 166.1310, Accuracy: 0.1407, Learning rate: 0.10000
Epoch: 4/5 (14.9285 s): Loss: 109.9386, Accuracy: 0.1859, Learning rate: 0.10000
Epoch: 5/5 (15.1280 s): Loss: 158.9268, Accuracy: 0.1859, Learning rate: 0.10000


For these examples the epochs are just 5 but incrementing the epochs the results dont change, tanh and relu for me perform worse than sigmoid.

Here is my python code reference for SGD:

This method was created to accept different activation functions to dynamically use them when creating the neural network object

The activation functions and their derivatives:

Activation functions and derivatives

The loss function i used is the mean squared error:

def mean_squared(output, expected_result):
return numpy.sum((output - expected_result) ** 2) / expected_result.shape[0]

def mean_squared_derivative(output, expected_result):
return output - expected_result



Is there some concept i am missing? Am i using the activation functions the wrong way? I really cannot find the answer to this even after searching for a long time. I feel like the problem is somewhere in the backpropagation but i can't find it. Any kind of help would be greatly appriciated

PS: I hope i posted this in the right place, i am pretty new to asking questions here, so if there is any problem i will move the question somewhere else

## Edit:

I tried to implement this with tensorflow, using relu for hidden layers and sigmoid for output. The results i get with this implementation are the same as the ones i mentioned in my question, so unless i am doing something wrong in both situations i am left to think i cannot use relu with sigmoid, which makes sense cause relu can have very high values while sigmoid pushes them down between 0 and 1, therefore most of the times giving values very close to 1.
Code reference:
TensorFlow implementation

• Did you try a lower learning rate for relu ? Apr 7 '20 at 22:18
• I tried any kind of learning rate, from 0.000001 to 10, also with schedules, but nothing improves Apr 7 '20 at 22:21