What are the mathematical prerequisites for understanding the core part of various algorithms involved in artificial intelligence and developing one's own algorithms?
Please, refer to some specific books.
$\begingroup$
$\endgroup$
0
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
Good Mathematics Foundation
Begin by ensuring full competency with intermediate algebra and some other foundations of calculus and discrete math, including the terminology and basic concepts within these topics.
- Infinite series
- Logical proofs
- Linear algebra and matrices
- Analytic geometry, especially the distinction between local and global extremes (minima and maxima), saddle points, and points of inflection
- Set theory
- Probability
- Statistics
Foundations of Cybernetics
Norbert Wiener, Cybernetics, 1948, MIT Press, contains time series and feedback concepts with clarity and command not seen in subsequent works; it also contains an introduction to information theory beginning with Shannon's log2 formula for a defining the amount of information in a bit. This is important to understand the expansion of the information entropy concept.
Calculus
Find a good calculus book and make sure you have clarity around key theory and application in these categories.
- Time series
- Infinite series
- Convergence — Artificial networks ideally converge to an optimum during learning.
- Partial differentials
- Jacobian and Hessian matrices
- Multivariate math
- Boundary regions
- Discrete math
Much of that is in Calculus, Strang, MIT, Wellesley-Cambridge Press. Although the PDF is available on the web, it is basic and not particularly deep. The one in our laboratory's library is Intermediate Calculus, Hurley, Holt Rinehart & Winston, 1980. It is comprehensive and in some ways better laid out than the one I have in my home library, which Princeton uses for sophomores.
Ensure you are comfortable working in spaces beyond ℝ2 (beyond 2D). For instance, RNNs are often in spaces such as ℝ4 thorugh ℝ7 because of the horizontal, vertical, pixel depth, and movie frame dimensions.
Finite Math
It is unfortunate that no combination of any three books I can think of has all of these.
- Directed graphs — Learn this BEFORE trees or circuits (artificial nets) because it is the superset topography of all those configurations
- Abstract symbol trees (ASTs)
- Advanced set theory
- Decision trees
- Markov chains
- Chaos theory (especially the difference between random and pseudo-random)
- Game Theory starting with Von Neumann and Morgenstern's Game Theory, the seminal work in that field
- Convergence in discrete systems especially the application of theory to signal saturation in integer, fixed point, or floating-point arithmetic
- Statistical means, deviations, correlation, and the more progressive concepts of entropy, relative entropy, and cross-entropy
- Curve fitting
- Convolution
- Probability especially Bayes' Theorem
- Algorithmic theory (Gödel's uncertainty theorems and Turing completeness)
Chemistry and Neurology
It is good to recall chemical equilibria from high school chemistry. Balance plays a key role in more sophisticated AI designs. Understanding the symbiotic relationship between generative and discriminative models in GANs will help a student further this understanding.
The control functions within biological systems remain a primary source of proofs of concept in artificial intelligence research. As researchers become more creative in imagining forms of adaptation that do not directly mimic some aspect of biology (still a distance off as of this writing) creativity may play a larger role in AI research objective formulation.
Even so, AI will probably remain a largely interdisciplinary field.
-
2$\begingroup$ Some comments: 1) I agree with what John wrote in his answer, that his answer is about a more general "core", whereas yours includes things that may be useful or may not be depending on what area of AI someone gets into. 2) Many things you describe under "high school math" are not (necessarily) high school math, at least not in Europe (don't know about US). In Netherlands, I didn't really get any Linear Algebra, matrices, infinite series, or set theory until my first year in university. Some of them might have appeared earlier if I had chosen a different set of courses in high school though. $\endgroup$– Dennis Soemers ♦Commented Aug 7, 2018 at 9:50
-
2$\begingroup$ 3) Functional Analysis / Measure Theory may be useful to include in some areas. But, again, it very very much depends on how deep you want to go as an AI researcher. Some AI researchers on the more theoretical side of things will find almost all this stuff useful. Other AI researchers more on the empirical / software / programming side need much, much less. Both can still output highly valuable research. $\endgroup$– Dennis Soemers ♦Commented Aug 7, 2018 at 9:51
$\begingroup$
$\endgroup$
0
I work as a professor, and recently designed the mathematics requirements for a new AI major, in consultation with many of my colleagues at other institutions.
The other answers, particularly this one do a good job of cataloging all the specific topics that might be useful somewhere in AI, but not all of them are equally useful for understanding core topics. In other cases, understanding the topic is essentially the same as understanding the related AI algorithms, so we usually just teach them together instead of assuming prerequisite knowledge. For instance, Markov Decision Processes aren't hard to teach to someone who already knows the basics of graph theory and probabilities, so we usually just cover them when we teach reinforcement learning in an AI course, rather than as a separate topic in a mathematics course.
The mathematics requirements we settled on look like:
A one or two semester course in discrete mathematics. This is as much to establish comfort with proof and mathematical rigor as with any specific topic in the area. It's mostly just "foundational" knowledge, but bits of it turn out to be very useful. Comfort with infinite summations, the basics of graphs, combinatorics, and asymptotic analysis are perhaps the most directly applicable parts. I like Susanna Epp's book.
A one or two semester course in linear algebra, which is useful across a wide variety of topics in AI, especially machine learning and data mining. Lay & Lay is an okay book, but probably not the absolute best. Shilov is a recommendation from Ian Goodfellow and others, but I've not tried it myself.
A course in probability, and possibly a modern course in statistics (i.e. with a Bayesian focus). An older course in statistics, or one targeting social scientists, is not very useful though. My statistician colleagues are using Lock5 right now, and having good experiences with it.
At least differential and integral calculus, and preferably at least partial derivatives in vector calculus, but perhaps the whole course. This is useful in optimization, machine learning, and economics-based approaches to AI. Stewart is the most common textbook. It's comprehensive, and can be used for all three courses, but it's explanations aren't always the very best. I'd still recommend it though.
Those are the core topics. If you don't also have a traditional background in programming, then a course in graph theory and the basics of asymptotic complexity or algorithm design and analysis might be good supplements. Usually AI'ers come from a standard computer science background though, which covers all those things very well.
answered Aug 1, 2018 at 19:06
$\begingroup$
$\endgroup$
As far as simple algorithms like Gradient Descent are concerned, you need to have a good grasp of partial derivatives. Especially if you want to implement neural networks. Also most algorithms are vectorised to improve computing speed and so you need to be comfortable with matrix math. This involves being really quick and comfy with dimensions of matrices, dimensions of products, multiplication of matrices, transpose and so on. Very rarely, you might use matrix calculus to directly arrive at optimal solutions, so a few results from this area should do. Moving on, you need to understand some function analysis. this is needed to get an intuition on what activation functions like sigmoid and tanh, log are doing. A grasp of probability and expectations is also really useful. You should also be clear with orthogonal vectors and inner products.
That being said, I would suggest you grasp basic calculus and matrix operations and try learning AI concepts. If you can't figure something out, explore the math.
Note: again this is only for starting.
