I am reading the Goodfellow's book about neural networks, but I am stuck in the mathematical calculus of the back-propagation algorithm. I understood the principle, and some Youtube videos explaining this algorithm shown step-by-step, but now I would like to understand the matrix calculus, that is, calculus with matrices and vectors, but especially everything related to the derivatives with respect to a matrix or a vector, and so on.

Which math book could you advise me to read?

I specify I studied 2 years after the bachelor in math school (in French: mathématiques supérieures et spéciales), but did not practice since years.

  • $\begingroup$ Retro-propagation is a term from quantum neurology. The theoretical foundation of space and time can be found in mathematical books about physics. $\endgroup$ – Manuel Rodriguez Oct 28 '19 at 17:59

Linear Algebra Done Right by Axler seems to be the best book on linear algebra, with a brisk and modern approach.


If you already have two years of a bachelor's of mathematics, I recommend part I of Goodfellow et al.. Features:

  • This book is very recent.
  • This book is free.
  • This book reviews exactly the mathematics used in the optimization of neural nets (in part 1), and then actually goes through the various models in detail in the later parts. The review is done at a level that is suitable for someone who has already studied these topics, but needs a refresher.
  • $\begingroup$ I have just had a quick look at this book (of which I've only read a few pages), but it doesn't seem to go into the details of vectorized operations and matrix calculus. $\endgroup$ – nbro Oct 29 '19 at 0:29
  • $\begingroup$ Chapter 2 goes into matrix operations. Chapter 4 covers numerical methods and calculus for matrices. $\endgroup$ – John Doucette Oct 29 '19 at 0:33
  • $\begingroup$ Chapter 2 covers the basics of linear algebra, but (I think) not exactly (or at least not especially) what the OP is looking for, that is, the calculus (differentiation) that involves matrices and vectors, even though the title is a little bit misleading. $\endgroup$ – nbro Oct 29 '19 at 0:36
  • $\begingroup$ @nbro Check out chapter 4. $\endgroup$ – John Doucette Oct 29 '19 at 0:40
  • 1
    $\begingroup$ It describes stuff like gradient descent and related math, but I think the OP is looking for a general math book that goes into the details of matrix calculus (related to neural networks and machine learning), but maybe I am wrong. By the way, the OP is already talking about this book (and I guess he's aware of the first part of the book), so I am not sure why you're suggesting it. If he's not happy with it, it probably means that this is not a good suggestion. $\endgroup$ – nbro Oct 29 '19 at 0:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.