Quick Answer
When Intel acquired Nirvana, they indicated their belief that analog VLSI has its place in the neuromorphic chips of the near future1, 2, 3.
Whether it was because of the ability to more easily exploit the natural quantum noise in analog circuits is not yet public. It is more likely because of the number and complexity of parallel activation functions that can be packed into a single VLSI chip. Analog has orders of magnitude advantage over digital in that respect.
It is likely beneficial for AI Stack Exchange members to come up to speed on this strongly indicated evolution of technology.
Important Trends and Non-trends in AI
To approach this question scientifically, it is best to contrast analog and digital signal theory without the bias of trends.
Artificial intelligence enthusiasts can find much on the web about deep learning, feature extraction, image recognition, and the software libraries to download and immediately start experimenting. It is the way that most get their feet wet with the technology, but the fast track introduction to AI has its down side too.
When the theoretical foundations of early successful deployments of consumer-facing AI are not understood, assumptions form that conflict with those foundations. Important options, such as analog artificial neurons, spiked networks, and real time feedback, are overlooked. The improvement of forms, capabilities, and reliability are compromised.
Enthusiasm in technology development should always be tempered with at least an equal measure of rational thought.
Convergence and Stability
In a system where accuracy and stability are achieved through feedback, both analog and digital signal values are always mere estimates.
- Digital values in a converging algorithm, or, more precisely, an strategy designed to converge
- Analog signal values in a stable operational amplifier circuit
Understanding the parallel between convergence through error correction in a digital algorithm and stability achieved through feedback in analog instrumentation is important in thinking about this question. These are the parallels using contemporary jargon, with digital on the left and analog on the right.
┌───────────────────────────────┬──────────────────────────────┐
│ * Digital Artificial Nets * │ * Analog Artificial Nets * │
├───────────────────────────────┼──────────────────────────────┤
│ Forward propagation │ Primary signal path │
├───────────────────────────────┼──────────────────────────────┤
│ Error function │ Error function │
├───────────────────────────────┼──────────────────────────────┤
│ Convergent │ Stable │
├───────────────────────────────┼──────────────────────────────┤
│ Saturation of gradient │ Saturation at inputs │
├───────────────────────────────┼──────────────────────────────┤
│ Activation function │ Forward transfer function │
└───────────────────────────────┴──────────────────────────────┘
Popularity of Digital Circuits
The primary factor in the rise of digital circuit popularity is its containment of noise. Today's VLSI digital circuits have long mean times to failure (mean time between instances when an incorrect bit value is encountered).
The virtual elimination of noise gave digital circuitry a significant advantage over analog circuitry for measurement, PID control, calculation, and other applications. With digital circuitry, one could measure to five decimal digits of accuracy, control with remarkable precision, and calculate π to a thousand decimal digits of accuracy, repeatably and reliably.
It was primarily aeronautics, defense, ballistics, and countermeasures budgets that raised the manufacturing demand to achieve the economy of scale in digital circuit manufacture. Demand for display resolution and rendering speed is driving the GPU use as a digital signal processor now.
Are these largely economic forces causing the best design choices? Are digitally based artificial networks the best use of precious VLSI real estate? That is the challenge of this question, and it is a good one.
Realities of IC Complexity
As mentioned in a comment, it takes tens of thousands of transistors to implement in silicon an independent, reusable artificial network neuron. This is largely because of the vector-matrix multiplication leading into each activation layer. It only takes a few dozen transistors per artificial neuron to implement a vector-matrix multiplication and the layer's array of operational amplifiers. Operational amplifiers can be designed to perform functions such as binary step, sigmoid, soft plus, ELU, and ISRLU.
Digital Signal Noise from Rounding
Digital signaling is not free of noise because most digital signals are rounded and therefore approximations. Saturation of the signal in back-propagation first appears as the digital noise generated from this approximation. Further saturation occurs when the signal is always rounded to the same binary representation.
Rounding approximation occurs in digital signal processing whenever the result of multiplication or division cannot be perfectly represented in binary, which is almost every time. Only those numbers that can be represented in a finite series of powers of two can be represented perfectly in binary. In the case of floating point numbers, this still holds. Here, $v$ is a perfectly represented value, $e$ is the exponent, $k$ is the position of the fractional part, $n$ is the mantissa bit, and $N$ is the number of bits in the mantissa.
$v = \sum_{n = 0}^{N} 1_n \, 2^{\, k + e + N - n}$
Programmers sometimes encounter the effects of rounding in double or single precision IEEE floating point numbers when answers that are expected to be 0.2 appear as 0.20000000000001. One fifth cannot be represented with perfect accuracy as a binary number because 5 is not a factor of 2.
Science Over Media Hype and Popular Trends
Those engaged in scientific investigation and technology feasibility study should certainly have some awareness of economics, but the direction of technology should be driven by technical merit and utility. If those criteria are met, it is only a matter of time before the financial community is convinced by those merits The financial merit of theoretical physics appeared weak until $E = mc^2$. The financial merit of a global public network seemed weak when the Internet was called ARPANET, before Netcom first sold dial in public web access with an email address.
In machine learning as with many products of technology, there are four key quality metrics.
- Efficiency (which drives speed and economy of use)
- Reliability
- Accuracy
- Comprehensibility (which drives maintainability)
Sometimes, but not always, the achievement of one compromises another, in which case a balance must be struck. Gradient descent is a convergence strategy that can be realized in a digital algorithm that nicely balances these four, which is why it is the dominant strategy in multi-layer perceptron training and in many deep networks.
Those four things were central to early cybernetics work of Norbert Wiener prior to the first digital circuits in Bell Labs or the first flip flop realized with vacuum tubes. The term cybernetics is derived from the Greek κυβερνήτης (pronounced kyvernítis) meaning steersman, where the ruder and sails had to compensate for constantly changing wind and current and the ship needed to converge on the intended port or harbor.
The trend driven wiew of this question might surround the idea of whether VLSI can be accomplished to achieve economy of scale for analog networks, but the criteria given by its author is to avoid trend driven views. Even if that were not the case, as mentioned above, considerably fewer transistors are required to produce artificial network layers with analog circuitry than with digital. For that reason, it is legitimate to answer the question assuming that VLSI analog is very much feasible at a reasonable cost if attention was directed toward accomplishing it.
Analog Artificial Network Design
Analog artificial nets are being investigated all over the world, including the IBM/MIT joint venture, Intel's Nirvana, Google, the U.S. Air Force as early as 19925, Tesla, and many others, some indicated in the comments and the addendum to this question.
The interest in analog for artificial networks has to do with the number of parallel activation functions involved in learning can fit on a square millimeter of VLSI chip real estate. That depends largely on how many transistors are required. The attenuation matrices (the learning parameter matrices)4 require vector-matrix multiplication, which requires a large number of transistors and thus a significant chunk of VLSI real estate.
There must be five independent functional components in a basic multilayer perceptron network if it is to be available for fully parallel training.
- The vector-matrix multiplication that parametrizes the amplitude of forward propagation between the activation functions of each layer
- The retention of parameters
- The activation functions for each layer
- The retention of activation layer outputs to apply in back-propagation
- The derivative of activation functions for each layer
In analog circuitry, with the greater parallelism inherent in the method of signal transmission, 2 and 4 may not be necessary. Feedback theory and harmonic analysis will be applied to the circuit design, using a simulator like Spice.
To consider cost, an equation can predict with reasonable accuracy the cost of the VLSI product as a function of standard VLSI packaging cost $c_p$, the function representing cost as a function of production volume $c(\int r)$, the function of production rate as a function of time and cost $r(t, c)$, time $t$, cost itself, the widths of each network layer of index $i$ for $I$ layers $w_i$, the number of transistors per attenuator4 $\tau_p$, and the number of transistors per activation and its derivative circuits $\tau_a$ and $\tau_d$ respectively.
$c = c_p \; c(\int r(t, c) \, dt) \; \Big( \sum_{i = 0}^{I - 2} \, (\tau_p w_i w_{i-1} + \tau_a w_i + \tau_d w_i) + \tau_a w_{I-1} + \tau_d w_{I-1} \Big)$
For common values of these circuits in current analog integrated circuits, we have a cost for analog VLSI chips that converges over time to a value at least three orders of magnitude below that of digital chips with equivalent training parallelism.
Directly Addressing Noise Injection
The question states, "We are using gradients (Jacobian) or second degree models (Hessian) to estimate next steps in a convergent algorithm and deliberately adding noise [or] injecting pseudo random perturbations to improve convergence reliability by jumping out local wells in the error surface during convergence."
The reason pseudo random noise is injected into the convergence algorithm during training and in real time re-entrant networks (such as reinforcement networks) is because of the existence of local minima in the disparity (error) surface that are not the global minima of that surface. The global minima is the optimal trained state of the artificial network. Local minima may be far from optimal.
This surface illustrates the error function of parameters (two in this highly simplified case 6) and the issue of a local minima hiding the existence of the global minima. The low points in the surface represent minima at the critical points of local regions of optimum training convergence. 7,8
Error functions are simply a measure of the disparity between the current network state during training and the desired network state. During the training of artificial networks, the goal is to find the global minimum of this disparity. Such a surface exists whether the sample data is labeled or un-labelled and whether the training completion criteria is internal or external to the artificial network.
If the learning rate is small and the initial state is at the origin of the parameter space, the convergence, using gradient descent, will converge to the leftmost well, which is a local minimum, not the global minimum on the right.
Even if the experts initializing the artificial network for learning is clever enough to pick the mid-point between the two minima, the gradient at that point still slopes toward the left hand minimum, and the convergence will arrive at a non-optimal training state. If the optimality of training is critical, which it often is, training will fail to achieve production quality results.
One solution in use is to add entropy to the convergence process, which is often simply the injection of the attenuated output of a pseudo random number generator. Another solution less often used is to branch the training process and try the injection of a large amount of entropy in a second convergent process so that there is a conservative search and a somewhat wild search running in parallel.
It is true that quantum noise in extremely small analog circuits are have greater uniformity to the signal spectrum from its entropy than a digital pseudo-random generator and much fewer transistors are required to achieve the higher quality noise. Whether the challenges of doing so in VLSI implementations have been overcome is yet to be disclosed by research labs embedded in governments and corporations.
- Will such stochastic elements used to inject measured quantities of randomness to enhance training speed and reliability be adequately immune to external noise during training?
- Will they be sufficiently shielded from internal cross-talk?
- Will a demand arise that will lower the cost of VLSI manufacture sufficiently to reach a point of greater use outside highly funded research enterprises?
All three challenges are plausible. What is certain and also very interesting is how designers and manufacturers facilitate digital control of the analog signal pathways and activation functions to achieve high speed training.
Footnotes
[1] https://ieeexplore.ieee.org/abstract/document/8401400/
[2] https://spectrum.ieee.org/automaton/robotics/artificial-intelligence/analog-and-neuromorphic-chips-will-rule-robotic-age
[3] https://www.roboticstomorrow.com/article/2018/04/whats-the-difference-between-analog-and-neuromorphic-chips-in-robots/11820
[4] Attenuation refers to multiplication of a signal output from one actuation by a trainable perameter to provide an addend to be summed with others for the input to an activation of a subsequent layer. Although this is a physics term, it is often used in electrical engineering and it is the appropriate term to describe the function of the vector-matrix multiplication that achieves what, in less educated circles, is called weighting the layer inputs.
[5] http://www.dtic.mil/dtic/tr/fulltext/u2/a256621.pdf
[6] There are many more than two parameters in artificial networks, but only two are depicted in this illustration because the plot can only be comprehensible in 3-D and we need one of the three dimensions for the error function value.
[7] Surface definition:
$z = (x-2)^2 + (y-2)^2 + 60 - \frac {40} {\sqrt{1 + (y - 1.1)^2 + (x - 0.9)^2}} - \frac {40} {(1 + {((y - 2.2)^2 + (x - 3.1)^2)}^4)}$
[8] Associated gnuplot commands:
set title "Error Surface Showing How Global Optimum Can be Missed"
set xlabel "x"
set ylabel "y"
set pm3d at b
set ticslevel 0.8
set isosample 40,40
set xrange [0:4]
set yrange [0:4]
set nokey
splot (x-2)**2 + (y-2)**2 + 60 \
- 40 / sqrt(1+(y-1.1)**2+(x-0.9)**2) \
- 40 / (1+(y-2.2)**2+(x-3.1)**2)**4
