In some tweets about NeurIPS 2018, this video from NVIDIA appeared. At around 0.37, she says:
If you think about the current computations in our deep learning systems, they are all based on Linear Algebra. Can we come up with better paradigms to do multi-dimensional processing? can we do truly tensor-algebraic techniques in our tensor cores?
I was wondering what she is talking about. I'm not an expert so I'd like to understand better this specific point.
Both PR and Theoretically Valid
Although Anima Anandkumar's presentation is a puff piece for NVidia, her representation is not contrary to theory.
... next level [above NVidia's GPU success] ... means new algorithmic research. So, if you think about the current computations in our deep learning systems, they are all based on linear algebra. Can we come up with better paradigms to do multi-dimensional processing. Can we do truly tensor algebraic techniques in our tensor cores, and what kind of new architectures will this realize?
Tensors
A tensor is the extension of the idea of functions and vectors. Vectors emerged allow the encapsulation of multiple variables of the same type into a single unit to express multidimensional properties such as position or force. Variables that are not encapsulations of multiple variables are then referred to as scalars. When such multidimensional properties are compositions of functions rather than variables because their values are dependent upon other variables, they are vector fields.
To more cleanly model electromagnetism and gravity in mathematics, higher levels of variable and functional encapsulation were required. Scalar variables and functions that return scalar variables are thus tensors of rank one (1). Vectors and vector fields are tensors of rank two (2), and higher ranks are extensions of the pattern created by scalars and vectors. A matrix containing functions can be represented as a tensor of rank three (3), encapsulating its elements into a single unit.
Digital Processing and Integration Trend
CUDA cores, which are the signal processing units used in current generation NVidia GPUs (graphical processing units) can be used for production of two or three dimensional rendering or leveraged to produce parallel processing of signals through an artificial network. This is following the trend to delegate to digital circuitry the most significant bottlenecks in algorithm execution. VLSI (very large scale integration) technology is the logical result of this trend.
With all these, speed, size, energy thrift, and convenience are not really a shift in the original paradigm that started with Norbert Wiener, Claude Shannon, Alan Turing, John von Neumann, and others. In fact, computing is still catching up in several ways with the perspectives of these pioneers and a long way off from producing in VLSI common concepts in science. Software is the solution, which is why it is called soft, meaning flexible, not necessarily weak. However, flexibility sacrifices speed and capability, thus the trend above.
The Parallel Computing Challenge
Part of what Anima Anandkumar is stating that the signal paths in current VLSI processors are still a much lower level of abstraction than the ideas of mathematicians, physicists, and AI engineers. Scientific theory describes probability, statistical distribution, expectation, force, loss, gain, pain, reward, memory, momentum, semantics, combination, and correlation at a much higher level than current digital circuitry.
The use of sequential algorithms is the splaying out in time what could be a massively parallel operation. The serial algorithm limits the rate of processing. The finding of ways of dealing with things in parallel in mathematics can be done with a pencil. In computers, paralleling algorithms and finding parallel processing structures in VLSI form that are as flexible as software in some respects is much more challenging and thus far behind.
This is one problem that has been the focus of research in at least twenty large corporate and government labs since for half a century and has been the intention of VLSI from the onset. The work is not specific to NVidia. It is not a new problem and the approach to solving it has been along this paradigm.
The last seventy years of computing machinery development has been bringing hardware, operating systems, and software closer to the level of mathematical expressions that were twenty to two hundred years old. That may change, and everyone wants to ride the new wave.
Anima Anandkumar and her counterparts in IBM, Intel, Google, Microsoft, the U.S. Navy, Amazon, Alibaba, and the other corporate and government laboratories do not state (because it is either classified or company confidential) what they intend to do to further the parallelization of computing. Whatever they do along those particular lines would not be a paradigm shift but rather a next step along the current paradigm.
Enters Corporate Strategy
They would also not state what they may be doing that is not in that paradigm. They are constrained to only give hints without theoretical substance. If they were working on a chip that exhibited what the human brain exhibits when neurons grow and connect according to DNA based propensity, they wouldn't say that in a technically precise way. Corporate secrecy is part of the global economic game play, not tipping the hand. They have their game face on.
The idea of moving from linear to non-linear is a good public relations theme and when used for puff pieces is not conclusively technical. The pitch usually goes along this line.
What they did was very linear. We are moving into a non-linear space.
It is an attempt to claim that what was done was primitive and the impending game changing advancement is coming from the speaker and their people. Sometimes it works to create a temporary hike in the value of tradable securities, which that company needs as of this writing. If there is a true change in the game, it will be known when it is released. Those who have worked in laboratories for years know to wait until something is released that demonstrates, when the example code is configured and ran, what game change actually occurred, if any. Or they develop the game changer themselves, which is why it is a lab.
Ambiguity of 'Linear' Even in Mathematics
Also note that curved lines are still lines and the term linear can mean two things depending on the context.
Predicting Paradigm Shifts
The skepticism developed after watching tech company PR for some time does not necessarily dismiss game changing technology advancements of the past and potentials of the future. Any open source project team, individual, corporation, or government lab might do something that shifts the paradigm, usually over a period of years. Classic examples:
Computing examples:
Shifts do not need to be as far reaching and game changing as these to have an impact. Who will achieve the next with regard to AI concepts, information structure, algorithm, execution environment is not something very easy to predict. Consider those that lived before any of the above and try to imagine them trying to predict that the projects of Isaac Newton, Antoine Lavoisier, Michael Faraday, Nicola Tesla, Norbert Wiener, Claude Shannon, Ken Thompson, Dennis Ritchie, or any of the others were the seeds of the next paradigm shift.
There is research into analog artificial networks, neuromorphic hardware, semantic modelling, graph algorithms, and other potential game changers, each of which has impressive conceptual foundations and is discussed in some of the Q&A here. These are a few.
What of these may point to the beginnings of paradigm shift cannot be known, and even if an idea first posted here or referenced from here is the seed, it may not be known later. The multilayer perceptron may be the seed of future streets and highways being dominated by automated vehicles in 2090, but no one in a million people will realize in seventy years that the trend toward AV research was seeded by MLP enthusiasm from the first decade of this century.
Gaps to be Filled and NVidia is a Contendere
All this aside, we use NVidia hardware every day for robotics and analysis, so they have credibility resulting from past success. If they produced a chip that does something remarkably clever before Intel or the IBM-MIT collaboration, it would be a small surprise, but not it is not a completely unbelievable possibility. Certainly the comprehension of Hilbert spaces, semantics, and topology are limited in the computer science field, and a paradigm shift toward greater comprehension of them or some new thing that is not even part of mathematical thought today would add some needed diversity to the computing industry.
