Why did machine learning only become viable after Nvidia's chips were available?

Question

I listened to a talk attended by a panel consisting of two influential Chinese scientists: Wang Gang and Yu Kai, among others.

When asked about the biggest bottleneck in the development of artificial intelligence in the near future (3 to 5 years), Yu Kai, who has a background in the hardware industry, said that hardware would be the essential problem and that we should focus most of our attention on it. He gave us two examples:

1. In the early development of computers, we compared machines by their chips;
2. ML/DL, which has been very popular in recent years, would be almost impossible without the employment of Nvidia's GPUs.

The fundamental algorithms existed already in the 1980s and 1990s, but AI went through 3 AI winters and was not empirical until we could train models with GPU boosted mega servers.

Dr. Wang then commented on Yu's opinions, stating that we should also develop software systems because we cannot build an autonomous car even if we combine all the GPUs and computational power in the world.

Then, as usual, my mind wandered off, and I started thinking: what if those who could operate supercomputers in the 1980s and 1990s had utilized the then-existing neural network algorithms and trained them with tons of scientific data? Some people at that time could obviously attempt to build the AI systems we are building now.

But why did AI/ML/DL become a hot topic and practical only decades later? Is it merely a matter of hardware, software, and data?

Answer 1

There is a lot of factors for the boom of AI industry. What many people miss though is the boom has mostly been in the Machine Learning part of AI. This can be attributed to various simple reasons along with their comparisons during earlier times:

• Mathematics: The maths behind ML algorithms are pretty simple and known for a long time (whether it would work or not was not known though). During earlier times it was not possible to implement algorithms which require high precision of numbers, to be calculated on a chip, in an acceptable amount of time. One of the main arithmetic operations division of numbers still takes a lot of cycles in modern processors. Older processors were magnitude times slower than modern processors (more than 100x), this bottleneck made it impossible to train sophisticated models on contemporary processors.
• Precision: Precision in calculations is an important factor in ML algorithms. 32 bit precision in processor was made in the 80's and was probably commercially available in the late 90's (x86), but it was still hella slow than current processors. This resulted in scientists improvising on the precision part and the most basic Perceptron Learning Algorithm invented in the 1960's to train a classifier uses only $1$'s and $0$'s, so basically a binary classifier. It was run on special computers. Although, it is interesting to note that we have come a full circle and Google is now using TPU's with 8-16 bit accuracy to implement ML models with great success.
• Parallelization : The concept of parallelization of matrix operations is nothing new. It was only when we started to see Deep Learning as just a set of matrix operations we realized that it can be easily parallelized on massively parallel GPU's, still if your ML algorithm is not inherently parallel then it hardly matters whether you use CPU or GPU (e.g. RNN's).
• Data: Probably the biggest cause in the ML boom. The Internet has provided opportunities to collect huge amounts of data from users and also make it available to interested parties. Since an ML algorithm is just a function approximator based on data, therefore data is the single most important thing in a ML algorithm. The more the data the better the performance of your model.
• Cost: The cost of training a ML model has gone down significantly. So using a Supercomputer to train a model might be fine, but was it worth it? Super computers unlike normal PC's are tremendously resource hungry in terms of cooling, space, etc. A recent article on MIT Technology Review points out the carbon footprint of training a Deep Learning model (sub-branch of ML). It is quite a good indicator why it would have been infeasible to train on Supercomputers in earlier times (considering modern processors consume much lesser power and gives higher speeds). Although, I am not sure but I think earlier supercomputers were specialised in "parallel+very high precision computing" (required for weather, astronomy, military applications, etc) and the "very high precison part" is overkill in Machine Learning scenario.

Another important aspect is nowadays everyone has access to powerful computers. Thus, anyone can build new ML models, re-train pre-existing models, modify models, etc. This was quite not possible during earlier times,

All this factors has led to a huge surge in interest in ML and has caused the boom we are seeing today. Also check out this question on how we are moving beyond digital processors.

Answer 2

GPUs were ideal for AI boom because:

• They hit the right time

AI has been researched for a LONG time. Almost half a century. However, that was all exploration of how algorithms would work and look. When NVIDIA saw that the AI is about to go mainstream, they looked at their GPUs and realized that the huge parallel processing power, with relative ease of programing, is ideal for the era that is to be. Many other people realized that too.

• GPUs are sort of general purpose accelerators

GPGPU is a concept of using GPU parallel processing for general tasks. You can accelerate graphics, or make your algorithm utilize 1000s of cores available on GPU. That makes GPU awesome target for all kinds of use cases including AI. Given that they are already available and are not too hard to program, its ideal choice for accelerating AI algorithms.

Answer 3

Machine learning has been around since the 1960's. They had computers that were less intelligent than the palm-pilots of the 1990's, and they did machine learning.

What most folks call "Machine Learning" is deep neural networks like those that started getting competitive at vision-related tasks in the early 2010's (teens). The gpgpu hardware standing on video cards (I had friends using the amiga bit-blitter for the task in the early 90's, @TadeuszWestawic) opened up a relative jump in performance, the software to drive that hardware like the methods from Hinton were mature and effective, and then some of the networks started doing amazing things. Personally I consider Giraffe (Lai, 2015) and AlexNet (Krizhevsky, 2012) to have been watershed moments that motivated the business folks to put resources into the field.

Answer 4

The year 2012 is also generally considered the start of the “deep learning revolution”. The term “deep learning” refers to a branch of ML that is based on neural networks with many layers (hence the term “deep”). Although this basic technology had been around for many years, it was in 2012 when [KSH12] used deep neural networks (DNNs) to win the ImageNet image classification challenge by such a large margin that it caught the attention of the wider community. Related advances on other hard problems, such as speech recognition, appeared around the same time (see e.g., [Cir+10; Cir+11; Hin+12]). These breakthroughs were enabled by advances in hardware technology (in particular, the repurposing of fast graphics processing units (GPUs) from video games to ML), data collection technology (in particular, the use of crowd sourcing tools, such as Amazon’s Mechanical Turk platform, to collect large labeled datasets, such as ImageNet), as well as various new algorithmic ideas.

Reference: Probabilistic Machine Learning: An Introduction