1
$\begingroup$

I have been working on my own AI for a while now, trying to implemented SGD with momentum from scratch in python. After looking around and studying all the maths behind it, i finally managed to implement SGD in a neural network that i trained to recognize the classic MNIST digits dataset. As activation function i always used sigmoid for both hidden and output neurons, and everything seems to work more or less ok, but now i wanted to step it up a bit and try to let SGD operate with different activations, so i added 2 other functions to my code: relu and tanh. The behaviours that i expected based on articles, documentation and "tutorials" found online were:
tanh: Should be slightly better than sigmoid
relu: should be much better than sigmoid and tanh
(By better i mean faster or at least higher accuracy the the end, or a mix of both)

Using tanh it looks like it's much slower converging to a minimum compared to sigmoid

Using relu...well, the results were very, VERY horrible Here's the outputs with the different activations (Learning rate: 0.1, Epochs: 5, MiniBatch size: 10, Momentum: 0.9)

Sigmoid training

[Sigmoid for hidden layers, sigmoid for output layer]
Epoch: 1/5 (14.3271 s): Loss: 0.0685, Accuracy: 0.6231, Learning rate: 0.10000
Epoch: 2/5 (14.0060 s): Loss: 0.0503, Accuracy: 0.6281, Learning rate: 0.10000
Epoch: 3/5 (14.0081 s): Loss: 0.0482, Accuracy: 0.6382, Learning rate: 0.10000
Epoch: 4/5 (13.8516 s): Loss: 0.0471, Accuracy: 0.7085, Learning rate: 0.10000
Epoch: 5/5 (13.9411 s): Loss: 0.0374, Accuracy: 0.7990, Learning rate: 0.10000

Tanh training

[Tanh for hidden layers, sigmoid for output layer]
Epoch: 1/5 (13.7553 s): Loss: 0.3708, Accuracy: 0.4171, Learning rate: 0.10000
Epoch: 2/5 (13.7666 s): Loss: 0.2580, Accuracy: 0.4623, Learning rate: 0.10000
Epoch: 3/5 (13.5550 s): Loss: 0.2289, Accuracy: 0.4824, Learning rate: 0.10000
Epoch: 4/5 (13.7311 s): Loss: 0.2211, Accuracy: 0.5729, Learning rate: 0.10000
Epoch: 5/5 (13.6996 s): Loss: 0.2142, Accuracy: 0.5779, Learning rate: 0.10000

Relu training

[Relu for hidden layers, sigmoid for output layer]
Epoch: 1/5 (14.2100 s): Loss: 0.7725, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 2/5 (14.6218 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 3/5 (14.2116 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 4/5 (14.1657 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
Epoch: 5/5 (14.1427 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000

Another run with relu

Epoch: 1/5 (14.7391 s): Loss: 15.4055, Accuracy: 0.1658, Learning rate: 0.10000
Epoch: 2/5 (14.8203 s): Loss: 59.2707, Accuracy: 0.1709, Learning rate: 0.10000
Epoch: 3/5 (15.3785 s): Loss: 166.1310, Accuracy: 0.1407, Learning rate: 0.10000
Epoch: 4/5 (14.9285 s): Loss: 109.9386, Accuracy: 0.1859, Learning rate: 0.10000
Epoch: 5/5 (15.1280 s): Loss: 158.9268, Accuracy: 0.1859, Learning rate: 0.10000

For these examples the epochs are just 5 but incrementing the epochs the results dont change, tanh and relu for me perform worse than sigmoid.

Here is my python code reference for SGD:

SGD with momentum

This method was created to accept different activation functions to dynamically use them when creating the neural network object

The activation functions and their derivatives:

Activation functions and derivatives

The loss function i used is the mean squared error:

def mean_squared(output, expected_result):
    return numpy.sum((output - expected_result) ** 2) / expected_result.shape[0]


def mean_squared_derivative(output, expected_result):
    return output - expected_result

Is there some concept i am missing? Am i using the activation functions the wrong way? I really cannot find the answer to this even after searching for a long time. I feel like the problem is somewhere in the backpropagation but i can't find it. Any kind of help would be greatly appriciated

PS: I hope i posted this in the right place, i am pretty new to asking questions here, so if there is any problem i will move the question somewhere else

Edit:

I tried to implement this with tensorflow, using relu for hidden layers and sigmoid for output. The results i get with this implementation are the same as the ones i mentioned in my question, so unless i am doing something wrong in both situations i am left to think i cannot use relu with sigmoid, which makes sense cause relu can have very high values while sigmoid pushes them down between 0 and 1, therefore most of the times giving values very close to 1.
Code reference:
TensorFlow implementation

$\endgroup$
  • $\begingroup$ Did you try a lower learning rate for relu ? $\endgroup$ – SpiderRico Apr 7 at 22:18
  • $\begingroup$ I tried any kind of learning rate, from 0.000001 to 10, also with schedules, but nothing improves $\endgroup$ – Alessandro Valentino Apr 7 at 22:21
1
$\begingroup$

Could you post the pseudocode of your backpropagation algorithm? I recommend you start off as simple as possible (this includes your cost f(x), I would simply use Yexpected-Youtput) and see if it works and then continue adding things. If it's your first time with neural networks, I recommend you check this link out and you could also try practising the algorithms on a programming language like Octave/Matlab (it can be very efficient speed wise). Also check this question out (link). At the bottom there is a code example for the XOR problem. Please post the pseudocode of your code instead of just dumping it there. Finally, don't just copy paste algorithms into your code, you need to understand them.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hi, thanks for the reply. All the code inside my python project was written by me, i had to study the maths and everything behind it, so it took a lot of time, but everything worked well (with sigmoid). Apparently something is missing. I did start out with simple things and just then i passed to the mnist classification task. My problem now is that i dont get the expected results using other activations other than sigmoid, but with that, is working pretty neat so i was wondering if it was something specific to my code that was causing the error. I will edit my question with the pseudo code $\endgroup$ – Alessandro Valentino Apr 7 at 14:59
  • $\begingroup$ I have only tried using the sigmoid and the step function so I can't really help you with the other ones. But you can check this link: en.wikipedia.org/wiki/Activation_function . $\endgroup$ – david david Apr 7 at 15:15
  • $\begingroup$ So, i tried using Yexpected-Youtput (as the cost derivative to calculate gradients) instead of Youtput-Yexpected. The results become as bad as relu with all activation functions $\endgroup$ – Alessandro Valentino Apr 8 at 6:44
  • $\begingroup$ As far as I know you don't need to find the derivative of the cost function. Only the activation function one. Check this link out if you are still unsure about backpropagation. I think that is your issue. home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html. You probably didn't convert the maths to the algorithms well enough and left something out $\endgroup$ – david david Apr 8 at 8:37
  • 1
    $\begingroup$ I'm sorry but if this is a problem with your code I can't really help you out. I don't know python. But anyway maybe you could plot a graph with your results after, I don't know, 500 epochs ... $\endgroup$ – david david Apr 8 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.