I'm trying to have a go at building a neural net, but I can't seem to figure out how to optimise the connections.

I've tried to have a look online and it came up with "backpropagation". I looked through some pages about it, but I can't seem to understand it.

It seems to be where you decide on a target value for each node of your output, and adjust the weights of the synapses to bring the values closer to their targets.

  • What about the inactive synapses (synapses giving a value of 0 because the previous neuron wasn't activated (its values didn't pass the threshold))? Do those stay the same?
  • How would this configure the hidden layers? Do I have to assign target values to them? How?
  • What are the other ways that the connections can be adjusted? What alternatives are there to using target values?

This is too broad a topic to answer directly.

If you are at the beginner stage with neural networks, you will need to learn some basic theory of the maths of neural networks, before the code will make sense. Although it is possible to write neural network code with only a vague understanding of what is going on, it is not a great way to learn for the future, and more advanced neural network features will likely remain beyond your comprehension.

The maths for back propagation are not that difficult conceptually, it is literally just the Chain Rule from basic calculus applied repeatedly. You do this to get a gradient that tells you the direction that an error function would increase in, then take a small step in the opposite direction. Repeat this over time and the error should reduce.

Despite the simplicity once you already know it, there are a lot of moving parts to training a neural network. The formulae for back propagation include multiple symbols with different meanings, and typically indexed in at least 3 dimensions all at once. It can look like a wall of impenetrable maths, especially if you have got a bit rusty at basic calculus and matrix multiplication, and need to review it.

The answer is to take your time and study the basics carefully. There are many resources out there. Introductory material should cover:

  • Revision of basic linear algebra. Just vectors, matrices and how to add and multiply them.

  • Revision of basic calculus. Just differentiation and gradients. The chain rule is handy for later.

  • Linear regression and logistic regression models trained using gradient descent. These are introductory models that naturally lead to layered neural networks.

  • Neural networks and back propagation.

There are many courses and books that attempt to teach you these things. It is hard to say in general which one would work best for a particular student. The best thing to do would be to go to a learning resource that you already like, such as Coursera or EDX, and search for "neural networks beginner" or similar.

Here are a couple of courses that I can personally recommend. I cannot create a comprehensive list, so if these do not apply on a quick browse (or if the links finally go stale after a few years), then you should search for something more suitable bearing in mind the syllabus suggestions above:

  • Andrew Ngs' Machine Learning covers more than just neural networks, but the extra parts apply to other aspects of ML, such as sensible approaches to validating and testing that anyone building predictive models should get to know. Code exercises are in Matlab or Octave.

  • Andrew Ngs' Introduction to Deep Learning starts with basic regression, and moves very quickly to practical builds of neural networks using TensorFlow. It is part of a five course "Specialisation" that works up to advanced NN models developed in the last three years.

  • Geoffrey Hinton's Neural Networks for Machine Learning covers some historical models (such as perceptrons) and some more obscure models, and the material is hard to understand, but quite rewarding if you are up to it. Not generally recommended to beginners, but if you have solid maths and computing knowledge already, at degree level, but never looked at a neural network before, this might be the right style of course for you.

They all happen to be Coursera courses . . . I am not affiliated, it is just that the relevant courses that I have personally taken have been on that platform.


Just thought I'd give an answer myself; covering the 3rd point.

  • What are the other ways that the connections can be adjusted? What alternatives are there to using target values?

I took a look on YouTube, and found https://www.youtube.com/watch?v=VnwjxityDLQ
It's a only 5 minute video. The videos I was getting when I searched "Andrew​ Ng's ML course" in YouTube were over an hour.

It said that the values could be optimised by creating a large number of copies of a network that have different random starting weights. Make the networks perform their task, and then identify the neurons that did the best, create copies of the neural nets that did the best, and then delete the nets that did the worst. After that, randomly alter the weights of the copies slightly. You can then repeat this process until the values are optimised.

This seems to be a good solution in that it doesn't require target values; it only needs to know how well it did. However, this method seems like it would require more RAM, and may be unsuitable for some situations.

It would still be helpful to know the other methods of adjusting the weights.

  • 2
    $\begingroup$ Well that's one way of doing it, but i suggest you not to try it. $\endgroup$ – user9947 Sep 4 '18 at 2:34
  • 1
    $\begingroup$ this method is called a "genetic algorithm" $\endgroup$ – Jérémy Blain Sep 4 '18 at 7:30
  • 4
    $\begingroup$ You might use this method if your goal is to use neural networks in control systems or games. The go-to algorithm of this approach is probably NEAT, and works very nicely for simple controllers. It is great for a-life scenarios too where you can evolve simplified artificial creatures that can learn to walk, jump or swim. However, research into more complex environments tends to use reinforcement learning, which uses gradient methods. $\endgroup$ – Neil Slater Sep 4 '18 at 8:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.