-1
$\begingroup$

I'm trying to have a go at building a neural net, but I can't seem to figure out how to optimise the connections.

I've tried to have a look online and it came up with "backpropagation". I looked through some pages about it, but I can't seem to understand it.

It seems to be where you decide on a target value for each node of your output, and adjust the weights of the synapses to bring the values closer to their targets.

  • What about the inactive synapses (synapses giving a value of 0 because the previous neuron wasn't activated (its values didn't pass the threshold))? Do those stay the same?
  • How would this configure the hidden layers? Do I have to assign target values to them? How?
  • What are the other ways that the connections can be adjusted? What alternatives are there to using target values?
$\endgroup$

put on hold as too broad by nbro, Djib2011, respectful, malioboro, olinarr 2 days ago

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

3
$\begingroup$

This is too broad a topic to answer directly.

If you are at the beginner stage with neural networks, you will need to learn some basic theory of the maths of neural networks, before the code will make sense. Although it is possible to write neural network code with only a vague understanding of what is going on, it is not a great way to learn for the future, and more advanced neural network features will likely remain beyond your comprehension.

The maths for back propagation are not that difficult conceptually, it is literally just the Chain Rule from basic calculus applied repeatedly. You do this to get a gradient that tells you the direction that an error function would increase in, then take a small step in the opposite direction. Repeat this over time and the error should reduce.

Despite the simplicity once you already know it, there are a lot of moving parts to training a neural network. The formulae for back propagation include multiple symbols with different meanings, and typically indexed in at least 3 dimensions all at once. It can look like a wall of impenetrable maths, especially if you have got a bit rusty at basic calculus and matrix multiplication, and need to review it.

The answer is to take your time and study the basics carefully. There are many resources out there. Introductory material should cover:

  • Revision of basic linear algebra. Just vectors, matrices and how to add and multiply them.

  • Revision of basic calculus. Just differentiation and gradients. The chain rule is handy for later.

  • Linear regression and logistic regression models trained using gradient descent. These are introductory models that naturally lead to layered neural networks.

  • Neural networks and back propagation.

There are many courses and books that attempt to teach you these things. It is hard to say in general which one would work best for a particular student. The best thing to do would be to go to a learning resource that you already like, such as Coursera or EDX, and search for "neural networks beginner" or similar.

Here are a couple of courses that I can personally recommend. I cannot create a comprehensive list, so if these do not apply on a quick browse (or if the links finally go stale after a few years), then you should search for something more suitable bearing in mind the syllabus suggestions above:

  • Andrew Ngs' Machine Learning covers more than just neural networks, but the extra parts apply to other aspects of ML, such as sensible approaches to validating and testing that anyone building predictive models should get to know. Code exercises are in Matlab or Octave.

  • Andrew Ngs' Introduction to Deep Learning starts with basic regression, and moves very quickly to practical builds of neural networks using TensorFlow. It is part of a five course "Specialisation" that works up to advanced NN models developed in the last three years.

  • Geoffrey Hinton's Neural Networks for Machine Learning covers some historical models (such as perceptrons) and some more obscure models, and the material is hard to understand, but quite rewarding if you are up to it. Not generally recommended to beginners, but if you have solid maths and computing knowledge already, at degree level, but never looked at a neural network before, this might be the right style of course for you.

They all happen to be Coursera courses . . . I am not affiliated, it is just that the relevant courses that I have personally taken have been on that platform.

$\endgroup$
  • 2
    $\begingroup$ I have made this a Community Wiki and suggest others contribute beginner courses, tutorials and books. I suggest that to qualify that they are each self-contained, and proper courses/books usable by beginners to the subject, no links to explanations of just backprop part on a blog please. $\endgroup$ – Neil Slater Sep 3 '18 at 15:37
  • $\begingroup$ Why didn't you just vote to close this question as too broad? There a lot of comments that say that this question is too broad (and, of course, it is too broad!). Apparently, nobody knows that too broad questions can be closed, and should remain closed until they are narrowed down and thus fixed. The regular users, like you, should be well aware of this. $\endgroup$ – nbro Nov 16 at 18:05
  • $\begingroup$ @nbro: Because it is a commonly asked type of question, and for this question specifically I thought it best to frame challenge the direct answer to what was requested and provide resources that together do answer the OP's implied need. This is what my first sentence/paragraph is trying to establish - i.e. "I am not answering fully, but pointing you to a decent soruce of self-help" $\endgroup$ – Neil Slater Nov 16 at 20:43
  • $\begingroup$ It would have been better to advise the OP to ask one specific question about any of the doubts and advise him/her to post the other more specific questions in their separate post. $\endgroup$ – nbro Nov 16 at 21:35
2
$\begingroup$

Just thought I'd give an answer myself; covering the 3rd point.

  • What are the other ways that the connections can be adjusted? What alternatives are there to using target values?

I took a look on YouTube, and found https://www.youtube.com/watch?v=VnwjxityDLQ
It's a only 5 minute video. The videos I was getting when I searched "Andrew​ Ng's ML course" in YouTube were over an hour.

It said that the values could be optimised by creating a large number of copies of a network that have different random starting weights. Make the networks perform their task, and then identify the neurons that did the best, create copies of the neural nets that did the best, and then delete the nets that did the worst. After that, randomly alter the weights of the copies slightly. You can then repeat this process until the values are optimised.

This seems to be a good solution in that it doesn't require target values; it only needs to know how well it did. However, this method seems like it would require more RAM, and may be unsuitable for some situations.

It would still be helpful to know the other methods of adjusting the weights.

$\endgroup$
  • 2
    $\begingroup$ Well that's one way of doing it, but i suggest you not to try it. $\endgroup$ – DuttaA Sep 4 '18 at 2:34
  • 1
    $\begingroup$ this method is called a "genetic algorithm" $\endgroup$ – Jérémy Blain Sep 4 '18 at 7:30
  • 4
    $\begingroup$ You might use this method if your goal is to use neural networks in control systems or games. The go-to algorithm of this approach is probably NEAT, and works very nicely for simple controllers. It is great for a-life scenarios too where you can evolve simplified artificial creatures that can learn to walk, jump or swim. However, research into more complex environments tends to use reinforcement learning, which uses gradient methods. $\endgroup$ – Neil Slater Sep 4 '18 at 8:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.