For a graduate school project, I'm crafting the Neural Netowrk that I breifly talked about in the title. I've been working on it for a while. This might sound contradictory, but I'm very happy with how I've designed my code... although it does not train the weights correctly. I'm unsure as to why it doesn't. I'm able to customize the amount of nodes in each layer, and amount of layers in the program to be whatever I wish and it will compute an answer. But the answer is always off. I believe the answer to be something that I am structurally not understanding about the Sigmoid activation function, or gradient descent, and fixing the issue won't be something that fundamentally changes how my program is designed algorithmically.

Anyway, I'm asked to have a training set as an input, as well as "correct output", since this is supervised learning (the correct output being very simple, 2*x, where x is the input) that will train the weights around said function. It will not work for the life of me. For example, with an epoch of 50000, ill input data points between 0 and .5, with all of the output values being double that (since the function is 2x, so they will all be between .5 and 1). The final output will be nowhere close to correct, and the weights aren't trained as they should. I believe it to be because the task I've been given might not be entirely possible, I'm interpreting the output data wrong, or the training set I'm using is bad. my advisor wants me to use this activation function to do this task, and I don't want to disappoint.

I believe the answer involves clarification on something that is pretty structural in the understanding of deep learning/sigmoid functions, or what have you. So I didn't go in too much detail when it comes to my design since I don't expect the answer to be a specific issue with my design. However, I'd be more than happy to give more details should someone ask for clarification. Apologies if my post isn't worded too well. Thank you.

  • $\begingroup$ A hand-written neural network not learning correctly can have a lot of causes sadly, so it is not really possible to analyse your problem without more details. Could be a bug, a misunderstanding about theory, or poor choice of hyperparameters for the problem. You might be able to find similar questions here and on Data Science stack exchange where someone explains how to fix a specific broken NN (I've written a few of those answers). $\endgroup$ – Neil Slater Feb 23 at 12:32
  • $\begingroup$ In terms of being happy with the code, have you implemented any unit tests around forward propagation, gradient calculations etc? A good unit test around each sub-unit of your NN code will help you rule out a problem in that part. You could start to write some now, in order to double-check individual pieces. Gradient checking is similarly worthwhile, and could be written as an automated test too $\endgroup$ – Neil Slater Feb 23 at 12:34
  • $\begingroup$ Hi. Could you please put your main specific question in the title (rather than saying "I am doing X but I am stuck")? Of course, if you created a post, it means that you have a problem and hopefully a question, so just put that question in the title. Thanks. $\endgroup$ – nbro Feb 24 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.