0
$\begingroup$

I'm trying to have a go at building a neural net, but I can't seem to figure out how to optimise the connections.

I've tried to have a look online and it came up with "backpropagation". I looked through some pages about it, but I can't seem to understand it.

It seems to be where you decide on a target value for each node of your output, and adjust the weights of the synapses to bring the values closer to their targets.

  • What about the inactive synapses (synapses giving a value of 0 because the previous neuron wasn't activated (its values didn't pass the threshold))? Do those stay the same?
  • How would this configure the hidden layers? Do I have to assign target values to them? How?
  • What are the other ways that the connections can be adjusted? What alternatives are there to using target values?
$\endgroup$
  • 1
    $\begingroup$ You are asking a very broad question..I suggest you go through Andre Ng's ML course and then ask the details. $\endgroup$ – DuttaA Sep 3 '18 at 12:35
  • 2
    $\begingroup$ Like I said you should check out Andrew​ Ng's ML course... Answering this question properly would require lengthy mathematical statements along with formulas.. You'll find the course on YouTube or Coursera..It's a one stop course for beginners as well as experts who need a revision on some basic concepts $\endgroup$ – DuttaA Sep 3 '18 at 12:48
  • 2
    $\begingroup$ @DuttaA: Rather than closing this one immediately, do you think it could be a canonical question we could use for duplicate closure: "If you are a complete beginner, then take a course on NNs. Here are some good ways to find a suitable course, and (maybe) here are some beginner NN courses." $\endgroup$ – Neil Slater Sep 3 '18 at 13:48
  • 2
    $\begingroup$ @DuttaA: Here's what I suggest: We answer this one as well as possible. Maybe close it afterwards, because it is still off-topic, just it is such a common beginner question that we'd like somewhere to point people quickly in future. Then when similar question occurs, we close it quickly with "duplicate of..." close reason, pointing at this one. The reason I suggest this question is that it is mostly well-written and includes terms like "back propagation". I agree though that "inactive synapses" is not a usual term in NNs. $\endgroup$ – Neil Slater Sep 3 '18 at 14:17
  • 1
    $\begingroup$ @SuperS: It is off topic because it is too broad. It is not possible to write an answer on the site which is essentially a beginner's tutorial for neural networks. Doubly so if you have read some articles already and found them too much - clearly we'd need to write more than is in those articles to give you the right grounding. The topic is not complicated once you get to know it, but the amount of background material you need too much for a Q&A. Typically a couple of hours of lectures and a few pages of notes will cover it assuming you have vague memories of calculus and linear algebra . . $\endgroup$ – Neil Slater Sep 3 '18 at 14:21
3
$\begingroup$

This is too broad a topic to answer directly.

If you are at the beginner stage with neural networks, you will need to learn some basic theory of the maths of neural networks, before the code will make sense. Although it is possible to write neural network code with only a vague understanding of what is going on, it is not a great way to learn for the future, and more advanced neural network features will likely remain beyond your comprehension.

The maths for back propagation are not that difficult conceptually, it is literally just the Chain Rule from basic calculus applied repeatedly. You do this to get a gradient that tells you the direction that an error function would increase in, then take a small step in the opposite direction. Repeat this over time and the error should reduce.

Despite the simplicity once you already know it, there are a lot of moving parts to training a neural network. The formulae for back propagation include multiple symbols with different meanings, and typically indexed in at least 3 dimensions all at once. It can look like a wall of impenetrable maths, especially if you have got a bit rusty at basic calculus and matrix multiplication, and need to review it.

The answer is to take your time and study the basics carefully. There are many resources out there. Introductory material should cover:

  • Revision of basic linear algebra. Just vectors, matrices and how to add and multiply them.

  • Revision of basic calculus. Just differentiation and gradients. The chain rule is handy for later.

  • Linear regression and logistic regression models trained using gradient descent. These are introductory models that naturally lead to layered neural networks.

  • Neural networks and back propagation.

There are many courses and books that attempt to teach you these things. It is hard to say in general which one would work best for a particular student. The best thing to do would be to go to a learning resource that you already like, such as Coursera or EDX, and search for "neural networks beginner" or similar.

Here are a couple of courses that I can personally recommend. I cannot create a comprehensive list, so if these do not apply on a quick browse (or if the links finally go stale after a few years), then you should search for something more suitable bearing in mind the syllabus suggestions above:

  • Andrew Ngs' Machine Learning covers more than just neural networks, but the extra parts apply to other aspects of ML, such as sensible approaches to validating and testing that anyone building predictive models should get to know. Code exercises are in Matlab or Octave.

  • Andrew Ngs' Introduction to Deep Learning starts with basic regression, and moves very quickly to practical builds of neural networks using TensorFlow. It is part of a five course "Specialisation" that works up to advanced NN models developed in the last three years.

  • Geoffrey Hinton's Neural Networks for Machine Learning covers some historical models (such as perceptrons) and some more obscure models, and the material is hard to understand, but quite rewarding if you are up to it. Not generally recommended to beginners, but if you have solid maths and computing knowledge already, at degree level, but never looked at a neural network before, this might be the right style of course for you.

They all happen to be Coursera courses . . . I am not affiliated, it is just that the relevant courses that I have personally taken have been on that platform.

$\endgroup$
  • 2
    $\begingroup$ I have made this a Community Wiki and suggest others contribute beginner courses, tutorials and books. I suggest that to qualify that they are each self-contained, and proper courses/books usable by beginners to the subject, no links to explanations of just backprop part on a blog please. $\endgroup$ – Neil Slater Sep 3 '18 at 15:37
2
$\begingroup$

Just thought I'd give an answer myself; covering the 3rd point.

  • What are the other ways that the connections can be adjusted? What alternatives are there to using target values?

I took a look on YouTube, and found https://www.youtube.com/watch?v=VnwjxityDLQ
It's a only 5 minute video. The videos I was getting when I searched "Andrew​ Ng's ML course" in YouTube were over an hour.

It said that the values could be optimised by creating a large number of copies of a network that have different random starting weights. Make the networks perform their task, and then identify the neurons that did the best, create copies of the neural nets that did the best, and then delete the nets that did the worst. After that, randomly alter the weights of the copies slightly. You can then repeat this process until the values are optimised.

This seems to be a good solution in that it doesn't require target values; it only needs to know how well it did. However, this method seems like it would require more RAM, and may be unsuitable for some situations.

It would still be helpful to know the other methods of adjusting the weights.

$\endgroup$
  • 2
    $\begingroup$ Well that's one way of doing it, but i suggest you not to try it. $\endgroup$ – DuttaA Sep 4 '18 at 2:34
  • 1
    $\begingroup$ this method is called a "genetic algorithm" $\endgroup$ – Jérémy Blain Sep 4 '18 at 7:30
  • 4
    $\begingroup$ You might use this method if your goal is to use neural networks in control systems or games. The go-to algorithm of this approach is probably NEAT, and works very nicely for simple controllers. It is great for a-life scenarios too where you can evolve simplified artificial creatures that can learn to walk, jump or swim. However, research into more complex environments tends to use reinforcement learning, which uses gradient methods. $\endgroup$ – Neil Slater Sep 4 '18 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.