Finding an optimum back propagation algorithm

Question

I recently started working on very simple machine learning codes in Python and I came across a big problem: teaching the system to improve on its guesses.

So this is what the code is about: I will have a list of organisms with their features stated in numerical values. I want to write a code that identifies that whether the organism is a cat or a fish or neither based on their characteristics. (For example an organism with a high fur value and 4 legs is more probable to be a cat.)

My idea for the neural network is to have five input nodes(for the five characteristics) and 2 output nodes (one for how cat it is and one for how fish it is). The input nodes are multiplied by a weight value and then all summed together to produce one of the output nodes. This repeats itself for the other output. How much the system got wrong is just the difference between the value of the output nodes and how cat/fish the being actually is.

But how can I use this information to correct the weights of the input nodes? Since the weights are randomly generated they can be in the "wrong" directions to begin with. For example if the subject is a cat then we should be expecting high fur and leg values. But what if the weight for the leg value is negative while the weight for the fur value is positive? Adding or multiplying the weight by the error won't bring us any closer to accurately determining the being. Is my neural network flawed to begin with? Or is there a rule of thumb in choosing back propagation algorithms?

Thanks.

Answer 1

When you are training a neural network, you use an algorithm called back propagation. This algorithm uses partial derivatives to determine the optimal values for weights. Partial derivatives are a calculus based method which tell you how far you need to adjust the weights in order to get to an optimum value. However, when you have neural networks with many different weights you need to be able to decide which ones to update in order to achieve a more optimal network. This is where stochastic gradient descent comes in. This algorithm is able to take many different gradients and traverse through many dimensional space into a lower error rate. Also, from your diagram of your neural network, I do not see any hidden neurons, which are neurons between the input and output layers which do most of the actual work. While you might be able to train your neural network optimally without hidden neurons, it might improve your accuracy depending upon your dataset. Finally, implementing a back propagation algorithm is a lot of work which is why many people use pre built libraries. Unless you are doing research, I recommend you do the same. I think for a problem like this, pybrain would be perfect, and if you want to use deep neural networks, tensorflow would be good for that.