I'm interested in learning about Neural Networks and implementing them. I'm particularly interested in GANs and LSTM networks.

I understand perceptrons and basic Neural Network configuration (sigmoid activation, weights, hidden layers etc). But what topics do I need to learn in order, to get to the point where I can implement GAN or LSTM.

I intend to make an implementation of each in C++ to prove to myself that I understand. I haven't got a particularly good math background, but I understand most math-things when they are explained.

For example, I understand backpropagation, but I don't really understand it. I understand how reinforced learning is used with backpropagation, but not fully how you can have things like training without datasets (like tD-backgammon). I don't quite understand CNNs, especially why you might make a particular architecture.

If for each "topic" there was a book or website or something for each it would be great.

  • $\begingroup$ For CNNs, use the cs231n stanford lectures on youtube - they're free. They teach back propagation very well in those lectures as well. As a recommendation, maybe try making your models in python, it will be a lot faster then c++, and the benefit of speed that c++ gives doesn't make sense in the context, as it will be very hard to implement a model as efficiently as Tensorflow is in c++. For an LSTM, I found information from this tutorial nice and concise medium.com/@aidangomez/let-s-do-this-f9b699de31d9, although the structure is drawn a bit differently. $\endgroup$
    – Recessive
    Oct 2, 2019 at 3:14
  • $\begingroup$ Many people will try to draw the full back prop as a series of equations, like this: miro.medium.com/max/1920/1*PBu6tUmPMxBqxEpZHujfPA.jpeg (you need to copy paste the full link) but please keep in mind this is over-complicating a rather simple algorithm once you come to understand it. If you get the chain rule, and the basic multi-variable calculus rule that f'(a,b) = f'(a) + f'(b) at points where gradients meet, you can do it entirely visually with no need for huge long strings of equations, which is how it's mostly done in the example I gave earlier. $\endgroup$
    – Recessive
    Oct 2, 2019 at 3:18
  • $\begingroup$ After understanding all of this for any model, you'll probably have the hardest time just finding the correct input and output dimensions of an architecture. A lot of articles either: don't understand the dimensions, or assume you already know them. CNNs are especially tricky, and I ended up programming one missing an entire dimension for filter depth (it still works fine, just learns a bit slower). It may take a while to learn any or all of this, but from experience it is very satisfying to finally see a model work and well worth the time. $\endgroup$
    – Recessive
    Oct 2, 2019 at 3:21
  • $\begingroup$ Hey thanks @Recessive. Is there a progression that you can think of that makes sense? I'm working through neuralnetworksanddeeplearning.com/… but was hoping to use more than one source $\endgroup$ Oct 2, 2019 at 3:35
  • $\begingroup$ Try to get to a position where you have learnt how to code from scratch a standard perceptron neural network. That tutorial looks good, although the back prop equations look a bit confusing. You only really need to know one for a sigmoid perceptron, and thats: wikimedia.org/api/rest_v1/media/math/render/svg/… where that represents the gradient at the end of each node going backwards (aka the delta) (from en.wikipedia.org/wiki/Backpropagation, keep in mind, sigmoid = logistic function). $\endgroup$
    – Recessive
    Oct 2, 2019 at 3:41

2 Answers 2


I think, once you are covered with the common stuff, you can probably go on and study all kinds of neural network variants.

The common stuff:

a) An undergraduate level Linear Algebra course -- covering matrix calculus. You might find this useful.

b) An undergraduate level study in statistical inference. Concepts from this topic will come up most of the time and you might have hard time getting around even though you understand the rest of the math. I would recommend this.

c) A starter book on neural networks. Ex- Neural networks by Raul Rojas.

After all these are covered you will certainly be ready for learning the variants of neural networks with ease. For LSTM I would recommend Alex Graves.


Would personally recommend deeplearning.ai's course to begin with. There may be more comprehensive or better MOOC's for covering basic MLP's, CNN, RNN's, tuning and training of neural networks but this is probably the most common one and the one that I can personally vouch for.

After this I'd recommend you get a physical or pdf copy of Deep Learning by Goodfellow et al. and use it as reference material for any new idea you'd want to learn. Personally would not recommend reading the whole book and its better as a reference material as it is quite comprehensive.

This should essentially be able to give you enough knowledge to be able to cover almost any paper/material on deep learning. The Course mentioned (most courses) would cover LSTM's as they are quite an old idea (~1997 I think) and GAN's are well covered in the book mentioned (The author invented them) since they are more of a recent advancement (2014).

Hope this was helpful!


Not the answer you're looking for? Browse other questions tagged .