What are the mathematical prerequisites for understanding the core part of the algorithms in artificial intelligence and developing own algorithm?
Please, refer me the specific books.
Good Mathematics Foundation
Begin by ensuring full competency with intermediate algebra and some other foundations of calculus and discrete math, including the terminology and basic concepts within these topics.
Foundations of Cybernetics
Norbert Wiener, Cybernetics, 1948, MIT Press, contains time series and feedback concepts with a clarity and command not seen in subsequent works; it also contains an introduction to information theory beginning with Shannon's log2 formula for a defining the amount of information in a bit. This is important to understand the expansion of the information entropy concept.
Find a good calculus book and make sure you have clarity around key theory and application in these categories.
Much of that is in Calculus, Strang, MIT, Wellesley-Cambridge Press. Although the PDF is available on the web, it is basic and not particularly deep. The one in our laboratory's library is Intermediate Calculus, Hurley, Holt Rinehart & Winston, 1980. It is comprehensive and in some ways better laid out than the one I have in my home library, which Princeton uses for sophomores.
Ensure you are comfortable working in spaces beyond ℝ2 (beyond 2D). For instance, RNNs are often in spaces such as ℝ4 thorugh ℝ7 because of the horizontal, vertical, pixel depth, and movie frame dimensions.
It is unfortunate that no combination of any three books I can think of has all of these.
Chemistry and Neurology
It is good to recall chemical equilibria from high school chemistry. Balance plays a key roll in more sophisticated AI designs. Understanding the symbiotic relationship between generative and discriminative models in GANs will help a student further this understanding.
The control functions within biological systems remain a primary source of proofs of concept in artificial intelligence research. As researchers become more creative in imagining forms of adaptation that do not directly mimic some aspect of biology (still a distance off as of this writing) creativity may play a larger roll in AI research objective formulation.
Even so, AI will probably remain a largely interdisciplinary field.
I work as a professor, and recently designed the mathematics requirements for a new AI major, in consultation with many of my colleagues at other institutions.
The other answers, particularly @FauChrisian's do a good job of cataloging all the specific topics that might be useful somewhere in AI, but not all of them are equally useful for understanding core topics. In other cases, understanding the topic is essentially the same as understanding the related AI algorithms, so we usually just teach them together instead of assuming prerequisite knowledge. For instance, Markov Decision processes aren't hard to teach to someone who already knows the basics of graph theory and probabilities, so we usually just cover them when we teach reinforcement learning in an AI course, rather than as a separate topic in a mathematics course.
The mathematics requirements we settled on look like:
A one or two semester course in discrete mathematics. This is as much to establish comfort with proof and mathematical rigor as with any specific topic in the area. It's mostly just "foundational" knowledge, but bits of it turn out to be very useful. Comfort with infinite summations, the basics of graphs, combinatorics, and asymptotic analysis are perhaps the most directly applicable parts. I like Susanna Epp's book.
A one or two semester course in linear algebra, which is useful across a wide variety of topics in AI, especially machine learning and data mining. Lay & Lay is an okay book, but probably not the absolute best. Shilov is a recommendation from Ian Goodfellow and others, but I've not tried it myself.
A course in probability, and possibly a modern course in statistics (i.e. with a Bayesian focus). An older course in statistics, or one targeting social scientists, is not very useful though. My statistician colleagues are using Lock5 right now, and having good experiences with it.
At least differential and integral calculus, and preferably at least partial derivatives in vector calculus, but perhaps the whole course. This is useful in optimization, machine learning, and economics-based approaches to AI. Stewart is the most common textbook. It's comprehensive, and can be used for all three courses, but it's explanations aren't always the very best. I'd still recommend it though.
Those are the core topics. If you don't also have a traditional background in programming, then a course in graph theory and the basics of asymptotic complexity or algorithm design and analysis might be good supplements. Usually AI'ers come from a standard computer science background though, which covers all those things very well.
As far as simple algorithms like Gradient Descent are concerned, you need to have a good grasp of partial derivatives. Especially if you want to implement neural networks. Also most algorithms are vectorised to improve computing speed and so you need to be comfortable with matrix math. This involves being really quick and comfy with dimensions of matrices, dimensions of products, multiplication of matrices, transpose and so on. Very rarely, you might use matrix calculus to directly arrive at optimal solutions, so a few results from this area should do. Moving on, you need to understand some function analysis. this is needed to get an intuition on what activation functions like sigmoid and tanh, log are doing. A grasp of probability and expectations is also really useful. You should also be clear with orthogonal vectors and inner products.
That being said, I would suggest you grasp basic calculus and matrix operations and try learning AI concepts. If you can't figure something out, explore the math.
Note: again this is only for starting.