What are other examples of theoretical machine learning books?

Question

I am looking for a book about machine learning that would suit my physics background. I am more or less familiar with classical and complex analysis, theory of probability, сcalculus of variations, matrix algebra, etc. However, I have not studied topology, measure theory, group theory, and other more advanced topics. I try to find a book that is written neither for beginners, nor for mathematicians.

Recently, I have read the great book "Statistical inference" written by Casella and Berger. They write in the introduction that "The purpose of this book is to build theoretical statistics (as different from mathematical statistics) from the first principles of probability theory". So, I am looking for some "theoretical books" about machine learning.

There are many online courses and brilliant books out there that focus on the practical side of applying machine learning models and using the appropriate libraries. It seems to me that there are no problems with them, but I would like to find a book on theory.

By now I have skimmed through the following books

• Pattern Recognition And Machine Learning

  It looks very nice. The only point of concern is that the book was published in 2006. So, I am not sure about the relevance of the chapters considering neural nets, since this field is developing rather fast.

• The elements of statistical learning

  This book also seems very good. It covers most of the topics as well as the first book. However, I am feeling that its style is different and I do not know which book will suit me better.

• Artificial Intelligence. A Modern Approach

  This one covers more recent topics, such as natural language processing. As far as I understand, it represents the view of a computer scientist on machine learning.

• Machine Learning A Probabilistic Perspective

  Maybe it has a slight bias towards probability theory, which is stated in the title. However, the book looks fascinating as well.

I think that the first or the second book should suit me, but I do not know what decision to make.

I am sure that I have overlooked some books.

Are there some other ML books that focus on theory?

Answer 1

Some of the books that you mention are often used as reference books in introductory courses to machine learning or artificial intelligence.

For example, if I remember correctly, in my introductory course to machine learning, the professor suggested the book Pattern Recognition And Machine Learning (2006) by Bishop, although we never used it during the lessons. This is a good book, but, in my opinion, it covers many topics, such as variational inference or sampling methods, that are not suited for an introductory course.

The book Artificial Intelligence. A Modern Approach, by Norvig and Russell, definitely does not focus on machine learning, but it covers many other aspects of artificial intelligence, such as search, planning, knowledge representation, machine learning, robotics, natural language processing or computer vision. This is probably the book that you should read and use if you want to have an extensive overview of the AI field. Although I never fully read it, I often used it as a reference, as I use the other mentioned book. For instance, during my bachelor's and, more specifically, an introductory course to artificial intelligence, we had used this book as the reference book, but note that there are other books that provide an extensive overview of the AI field.

The other two books are not as famous as these two, but they are probably also good books, although their focus may be different.

There are at least three other books that I think you should also be aware of, given that they also cover the actual theory of learning, aka (computational) learning theory, before diving into more specific topics, such as kernel methods.

• Machine Learning (1997) by Tom Mitchell
• Foundations of Machine Learning (2nd edition, 2018) by Mehryar Mohri et al.
• Understanding Machine Learning: From Theory to Algorithms (2014) by Shai Shalev-Shwartz et al.

You can find more books on learning theory here.

Answer 2

Pattern Recognition And Machine Learning is a great theoretical book. I don't know anything better on standard ML. I read several pages from it myself and all my colleagues researchers suggest to look there if you are not sure about some concepts. The 2 problems with it are that it's huge and it doesn't cover almost all deep learning models known for today.

So, in addition, I'd suggest you look at Deep Learning by Ian Goodfellow et al.

Your concerns about not studying topology, measure theory and group theory are groundless. These sections of math aren't prerequisites in any way, they aren't even discussed anywhere I know.

Actually, ML theory is more like probability theory and statistics. Especially, statistical learning theory (which is nothing more than probability theory and statistics). I haven't read any books on SLT so have a look at this answer.

Answer 3

For a more updated list I'd suggest these:

1. Deep Learning: Foundations and Concepts (2024) by Bishop (same author for the classic)
2. Reinforcement Learning: An Introduction by Rich Sutton
3. Multi-Agent Reinforcement Learning: Foundations and Modern Approaches (2024)
4. The Principles of Deep Learning Theory
5. Foundations of Computer Vision (2024)

There's also a list of 27 papers (reportedly shared by Ilya Sutskever with John Carmack).

"If you truly master these, you’ll understand 90% of what matters in modern AI."