I have read that all the math responsible for modern day machine learning and AI was already in place in 1900s but we did not have computational resources to implement those algorithms. So, is that true? And if it is, in what areas of machine learning the researchers work? And are all the future breakthroughs will be dependent only on increment of computational resources?
High-level answer: Increase in resources has been important in AI, and definitely was a factor with Deep Blue, but Machine Learning is a newer method that seems to produces more optimal results with less resources on problems of greater complexity.
Here is an article on AlphaGo's hardware: "Google reveals the mysterious custom hardware that powers AlphaGo"
Also an interesting analysis on Quora: "What hardware does AlphaGo run on? Is it customized hardware for best performance?"
Still pretty powerful systems, but I think the algorithms are as important as the computing resources, because all the hardware in the world won't help in the algorithms are poor.
Matthew Lai, creator of Giraffe Chess, said:
"In the ensuing two decades [since Deep Blue], both computer hardware and AI research advanced the state-of-art chess-playing computers to the point where even the best humans today have no realistic chance of defeating a modern chess engine running on a smartphone."
which suggests that hardware and software are both important parts of the overall equation.
most of the underlying math(like back-propergation) was discovered a long time ago, and the advances in hardware have only recently made it possible to tackle some problems within a reasonable amount of time but that is not the only reason. Some of the other things that also made deep learning pick up includes:
The availability of large, labeled, quality data - deep learning generally requires a lots of data to train successfully. This kind of data just wasn't there until recently.
Algorithmic improvements(I.e further improvements on the maths) - better weight initialization techniques, the discovery of the ReLu as a replacement for the sigmoid and tach activation which goes a long way towards solving the vanishing gradient problem, the discovery of neutral network dropout which is found to reduce overfitting, etc.
The discovery of better neutral network architectures.
So, whist the computing power is certainly a big part of it, it certainly isn't the only factor.