18
$\begingroup$

The impetus behind the twentieth century transition from analog to digital circuitry was driven by the desire for greater accuracy and lower noise. Now we are developing software where results are approximate and noise has positive value.

  • In artificial networks, we use gradients (Jacobian) or second degree models (Hessian) to estimate next steps in a convergent algorithm and define acceptable levels of inaccuracy and doubt.1
  • In convergence strategies, we deliberately add noise by injecting random or pseudo random perturbations to improve reliability by essentially jumping out local minima in the optimization surface during convergence.2

What we accept and deliberately introduce in current AI systems are the same things that drove electronics to digital circuitry.

Why not return to analog circuitry for neural nets and implement them with operational amplifier matrices instead of matrices of digital signal processing elements?

The values of artificial network learning parameters can maintained using integrated capacitors charged via D-to-A converters such that the learned states can benefit from digital accuracy and convenience, while forward propagation benefits from analog advantages.

  • Greater speed3
  • Orders of magnitude fewer transistors to represent network cells
  • Natural thermal noise4

An academic article or patent search for analog artificial networks reveals much work over the last forty years, and the research trend has been maintained. Computational analog circuits are well developed and provide a basis for neural arrays.

Could the current obsession with digital computation be clouding the common view of AI architectural options?

Is hybrid analog the superior architecture for artificial networks?

 


Footnotes

[1] The PAC (probably approximately correct) Learning Framework relates acceptable error $\epsilon$ and acceptable doubt $\delta$ to the sample size required for learning for specific model types. (Note that $1 - \epsilon$ represents accuracy and $1 - \delta$ represents confidence in this framework.)

[2] Stochastic gradient descent is shown, when appropriate strategies and hyper-parameters are used, to converge more quickly during learning and is becoming a best practice in typical real world applications of artificial networks.

[3] Intel Core i9-7960X Processor runs at turbo speeds of 4.2 GHz whereas the standard fixed-satelite broadcasting is 41 GHz.

[4] Thermal noise can be obtained on silicon by amplifying and filtering electron leakage across a reverse biased zener diodes at its avalanche point. The source of the quantum phenomena is Johnson–Nyquist thermal noise. Sanguinetti et. al. state in their 'Quantum Random Number Generation on a Mobile Phone' (2014), "A detector can be modeled as a lossy channel with a transmission probability η followed by a photon-to-electron converter with unit efficiency ... measured distribution will be the combination of quantum uncertainty and technical noise," and there's CalTech's JTWPA work. Both of these may become standards for producing truly nondeterministic quantum noise in integrated circuits.

References

$\endgroup$
  • 1
    $\begingroup$ I would argue that you're onto something. There's some efforts to put AI into analog chips (I think Apple might be doing something with iphone). I'm not sure how much research has been done but I'm sure you can find some white paper somewhere. It's definitely worth researching. My prediction is that there soon may be programmable AI chips that have a set number of inputs and outputs (Kinda like bus registers). $\endgroup$ – Zakk Diaz Sep 12 '18 at 17:49
  • $\begingroup$ It's not a full answer, but I suspect the main issue is cost. Printing circuits is super cheap at scale, and still pretty expensive in small batches. Discrete GPUs are mass produced already, and work "well enough". An analog chip usually can only do one task well, and the preferred models change quickly. A discrete chip can be programmed to do many different things. If we find a "best" topology for ANNs, maybe it will make sense to make analog chips again. $\endgroup$ – John Doucette Sep 12 '18 at 19:49
  • 1
    $\begingroup$ Wow. My first day on this site and I found someone who shares a thought with me. :-D $\endgroup$ – progyammer Sep 15 '18 at 9:12
6
$\begingroup$

I think, there are various reasons. First of all: Flexibility. With modern CPUs and GPUs you can construct pretty much every AI model you want and in every size and complexity you want. How can you be sure that the model you are currently using is still suitable in a few years? Maybe there will be a major breakthrough in NNs in the next few years? Maybe some scientists find that there is a way better way of developing an AI than with NNs, genetic algorithms etc. Normal chips can handle it all, and they can handle it well enough. But if you want to optimize it amd have no worries about money, you can develope a specialized architecture (this is already done by different companies, which gives a major speed boost on specific tasks).

Reason number two: Mass production. I mean, companies could eventually produce highly integrated analog AI components (let's say, for example NN chips). But that would be a bigger investment. It's rather unclear if units that are flexible enough to be a serious AI hardware alternative, can be easily produced in a mass-nm-production which can compete against CPUs and GPUs. Especially the latter are highly optimized for doing massive parallel calculation. And, if you watch the development of GPU-similar architectures (can do few things, but those very well) which are aditionally optimized for maschine learning, you can see that it would be a hard competition for analog units.

All the above doesn't mean that there is no research in this area. There a several experiments which try to archieve that, but they aren't 'dangerous' yet for common architectures. Eventually, they will come in the future, when we understand AI and intelligence in general better and are just trying to tweak, but I am rather skeptical about that.

EDIT: Also, something that also kinda belongs to flexibility: You can experiment better with the AI algorithms running on 'normal' digital hardware. For example, you can easily inspect a NN at certain locations, you can quickly modify the input data or provide alternative ones, you really aren't bound to anything. And since we still don't know or understand every model completely, when to use which, if there are better architectures for a certain task etc, it doesn't make sense to put something 'young' and 'experimental' in a fixed analog architecture.

$\endgroup$
  • $\begingroup$ Although the economy of scale (sheer manufacturing volume) favors digital today, it didn't in the 1980s and that many not in the 2040s. Analog is cheaper by transistor. There are 128,000 transistors in a CUDA core per thread and only 40 transistors in a multiplexed op amp. More importantly, the question is theoretical — what makes most technological sense — not what is economical at the current state of VLSI economics. If there is any pattern we can see in technology over the last 100 years is that today's normal is tomorrow's museum piece. — Reading the bounty requirements may help. $\endgroup$ – FauChristian Sep 13 '18 at 23:59
  • $\begingroup$ But isn't that kinda similar in this scenario? Massively developing that hardware NOW wouldn't make sense economical, but not technological either. We just don't know enough. $\endgroup$ – Ben Sep 14 '18 at 17:52
  • $\begingroup$ If, "we," is the AI Stack Exchange membership, there is a strong tendency toward whatever has already been implemented in popular Python libraries. But governments and large corporations seem to be interested in spiking networks and analog VLSI too, USAF and Intel for instance. There is a push from robotics labs toward analog, and neuro-cogitive researchers see ANNs as not worthy of the middle N. Real neuron are thousands of times more complex than a ReLU function. What will emerge as dominant for what application is unclear, but that's not the same as not knowing enough to discuss options. $\endgroup$ – FauChristian Sep 15 '18 at 4:40
  • $\begingroup$ You may have read the word "pure" into the question. None of the ongoing research suggests pure analog, with dials instead keyboards and CRTs instead of LCDs. All recent proposals in the literature and in active VLSI development follow a well understood paradigm: Simulate programmable (not fixed) analog that can learn the program as digital artificial networks can, then realize in silicon, without removing the programmability or the learning capacity. The real time signals may be analog, digital, or both, but the overall control of the chip is digital, as with a GPU or DSP. $\endgroup$ – FauChristian Sep 17 '18 at 20:41
  • $\begingroup$ The bounty period will soon end, and whether analog learning makes sense because it can capitalize on readily available quantum noise is not yet addressed in this answer. Prediction was not indicated by the question. Furthermore, the massive budget that seems to be targeted at analog computation of perceptrons, convolution, and spiking networks may very well prevail, but only if the long-term viability is rational. Thus the question. $\endgroup$ – FauChristian Sep 17 '18 at 20:47
6
+400
$\begingroup$

Quick Answer

When Intel acquired Nirvana, they indicated their belief that analog VLSI has its place in the neuromorphic chips of the near future1, 2, 3.

Whether it was because of the ability to more easily exploit the natural quantum noise in analog circuits is not yet public. It is more likely because of the number and complexity of parallel activation functions that can be packed into a single VLSI chip. Analog has orders of magnitude advantage over digital in that respect.

It is likely beneficial for AI Stack Exchange members to come up to speed on this strongly indicated evolution of technology.

Important Trends and Non-trends in AI

To approach this question scientifically, it is best to contrast analog and digital signal theory without the bias of trends.

Artificial intelligence enthusiasts can find much on the web about deep learning, feature extraction, image recognition, and the software libraries to download and immediately start experimenting. It is the way that most get their feet wet with the technology, but the fast track introduction to AI has its down side too.

When the theoretical foundations of early successful deployments of consumer-facing AI are not understood, assumptions form that conflict with those foundations. Important options, such as analog artificial neurons, spiked networks, and real time feedback, are overlooked. The improvement of forms, capabilities, and reliability are compromised.

Enthusiasm in technology development should always be tempered with at least an equal measure of rational thought.

Convergence and Stability

In a system where accuracy and stability are achieved through feedback, both analog and digital signal values are always mere estimates.

  • Digital values in a converging algorithm, or, more precisely, an strategy designed to converge
  • Analog signal values in a stable operational amplifier circuit

Understanding the parallel between convergence through error correction in a digital algorithm and stability achieved through feedback in analog instrumentation is important in thinking about this question. These are the parallels using contemporary jargon, with digital on the left and analog on the right.

┌───────────────────────────────┬──────────────────────────────┐
│  * Digital Artificial Nets *  │  * Analog Artificial Nets *  │
├───────────────────────────────┼──────────────────────────────┤
│  Forward propagation          │  Primary signal path         │
├───────────────────────────────┼──────────────────────────────┤
│  Error function               │  Error function              │
├───────────────────────────────┼──────────────────────────────┤
│  Convergent                   │  Stable                      │
├───────────────────────────────┼──────────────────────────────┤
│  Saturation of gradient       │  Saturation at inputs        │
├───────────────────────────────┼──────────────────────────────┤
│  Activation function          │  Forward transfer function   │
└───────────────────────────────┴──────────────────────────────┘

Popularity of Digital Circuits

The primary factor in the rise of digital circuit popularity is its containment of noise. Today's VLSI digital circuits have long mean times to failure (mean time between instances when an incorrect bit value is encountered).

The virtual elimination of noise gave digital circuitry a significant advantage over analog circuitry for measurement, PID control, calculation, and other applications. With digital circuitry, one could measure to five decimal digits of accuracy, control with remarkable precision, and calculate π to a thousand decimal digits of accuracy, repeatably and reliably.

It was primarily aeronautics, defense, ballistics, and countermeasures budgets that raised the manufacturing demand to achieve the economy of scale in digital circuit manufacture. Demand for display resolution and rendering speed is driving the GPU use as a digital signal processor now.

Are these largely economic forces causing the best design choices? Are digitally based artificial networks the best use of precious VLSI real estate? That is the challenge of this question, and it is a good one.

Realities of IC Complexity

As mentioned in a comment, it takes tens of thousands of transistors to implement in silicon an independent, reusable artificial network neuron. This is largely because of the vector-matrix multiplication leading into each activation layer. It only takes a few dozen transistors per artificial neuron to implement a vector-matrix multiplication and the layer's array of operational amplifiers. Operational amplifiers can be designed to perform functions such as binary step, sigmoid, soft plus, ELU, and ISRLU.

Digital Signal Noise from Rounding

Digital signaling is not free of noise because most digital signals are rounded and therefore approximations. Saturation of the signal in back-propagation first appears as the digital noise generated from this approximation. Further saturation occurs when the signal is always rounded to the same binary representation.

Rounding approximation occurs in digital signal processing whenever the result of multiplication or division cannot be perfectly represented in binary, which is almost every time. Only those numbers that can be represented in a finite series of powers of two can be represented perfectly in binary. In the case of floating point numbers, this still holds. Here, $v$ is a perfectly represented value, $e$ is the exponent, $k$ is the position of the fractional part, $n$ is the mantissa bit, and $N$ is the number of bits in the mantissa.

$v = \sum_{n = 0}^{N} 1_n \, 2^{\, k + e + N - n}$

Programmers sometimes encounter the effects of rounding in double or single precision IEEE floating point numbers when answers that are expected to be 0.2 appear as 0.20000000000001. One fifth cannot be represented with perfect accuracy as a binary number because 5 is not a factor of 2.

Science Over Media Hype and Popular Trends

Those engaged in scientific investigation and technology feasibility study should certainly have some awareness of economics, but the direction of technology should be driven by technical merit and utility. If those criteria are met, it is only a matter of time before the financial community is convinced by those merits The financial merit of theoretical physics appeared weak until $E = mc^2$. The financial merit of a global public network seemed weak when the Internet was called ARPANET, before Netcom first sold dial in public web access with an email address.

In machine learning as with many products of technology, there are four key quality metrics.

  • Efficiency (which drives speed and economy of use)
  • Reliability
  • Accuracy
  • Comprehensibility (which drives maintainability)

Sometimes, but not always, the achievement of one compromises another, in which case a balance must be struck. Gradient descent is a convergence strategy that can be realized in a digital algorithm that nicely balances these four, which is why it is the dominant strategy in multi-layer perceptron training and in many deep networks.

Those four things were central to early cybernetics work of Norbert Wiener prior to the first digital circuits in Bell Labs or the first flip flop realized with vacuum tubes. The term cybernetics is derived from the Greek κυβερνήτης (pronounced kyvernítis) meaning steersman, where the ruder and sails had to compensate for constantly changing wind and current and the ship needed to converge on the intended port or harbor.

The trend driven wiew of this question might surround the idea of whether VLSI can be accomplished to achieve economy of scale for analog networks, but the criteria given by its author is to avoid trend driven views. Even if that were not the case, as mentioned above, considerably fewer transistors are required to produce artificial network layers with analog circuitry than with digital. For that reason, it is legitimate to answer the question assuming that VLSI analog is very much feasible at a reasonable cost if attention was directed toward accomplishing it.

Analog Artificial Network Design

Analog artificial nets are being investigated all over the world, including the IBM/MIT joint venture, Intel's Nirvana, Google, the U.S. Air Force as early as 19925, Tesla, and many others, some indicated in the comments and the addendum to this question.

The interest in analog for artificial networks has to do with the number of parallel activation functions involved in learning can fit on a square millimeter of VLSI chip real estate. That depends largely on how many transistors are required. The attenuation matrices (the learning parameter matrices)4 require vector-matrix multiplication, which requires a large number of transistors and thus a significant chunk of VLSI real estate.

There must be five independent functional components in a basic multilayer perceptron network if it is to be available for fully parallel training.

  1. The vector-matrix multiplication that parametrizes the amplitude of forward propagation between the activation functions of each layer
  2. The retention of parameters
  3. The activation functions for each layer
  4. The retention of activation layer outputs to apply in back-propagation
  5. The derivative of activation functions for each layer

In analog circuitry, with the greater parallelism inherent in the method of signal transmission, 2 and 4 may not be necessary. Feedback theory and harmonic analysis will be applied to the circuit design, using a simulator like Spice.

To consider cost, an equation can predict with reasonable accuracy the cost of the VLSI product as a function of standard VLSI packaging cost $c_p$, the function representing cost as a function of production volume $c(\int r)$, the function of production rate as a function of time and cost $r(t, c)$, time $t$, cost itself, the widths of each network layer of index $i$ for $I$ layers $w_i$, the number of transistors per attenuator4 $\tau_p$, and the number of transistors per activation and its derivative circuits $\tau_a$ and $\tau_d$ respectively.

$c = c_p \; c(\int r(t, c) \, dt) \; \Big( \sum_{i = 0}^{I - 2} \, (\tau_p w_i w_{i-1} + \tau_a w_i + \tau_d w_i) + \tau_a w_{I-1} + \tau_d w_{I-1} \Big)$

For common values of these circuits in current analog integrated circuits, we have a cost for analog VLSI chips that converges over time to a value at least three orders of magnitude below that of digital chips with equivalent training parallelism.

Directly Addressing Noise Injection

The question states, "We are using gradients (Jacobian) or second degree models (Hessian) to estimate next steps in a convergent algorithm and deliberately adding noise [or] injecting pseudo random perturbations to improve convergence reliability by jumping out local wells in the error surface during convergence."

The reason pseudo random noise is injected into the convergence algorithm during training and in real time re-entrant networks (such as reinforcement networks) is because of the existence of local minima in the disparity (error) surface that are not the global minima of that surface. The global minima is the optimal trained state of the artificial network. Local minima may be far from optimal.

This surface illustrates the error function of parameters (two in this highly simplified case 6) and the issue of a local minima hiding the existence of the global minima. The low points in the surface represent minima at the critical points of local regions of optimum training convergence. 7,8

Error Surface Showing How Global Optimum Can be Missed

Error functions are simply a measure of the disparity between the current network state during training and the desired network state. During the training of artificial networks, the goal is to find the global minimum of this disparity. Such a surface exists whether the sample data is labeled or un-labelled and whether the training completion criteria is internal or external to the artificial network.

If the learning rate is small and the initial state is at the origin of the parameter space, the convergence, using gradient descent, will converge to the leftmost well, which is a local minimum, not the global minimum on the right.

Even if the experts initializing the artificial network for learning is clever enough to pick the mid-point between the two minima, the gradient at that point still slopes toward the left hand minimum, and the convergence will arrive at a non-optimal training state. If the optimality of training is critical, which it often is, training will fail to achieve production quality results.

One solution in use is to add entropy to the convergence process, which is often simply the injection of the attenuated output of a pseudo random number generator. Another solution less often used is to branch the training process and try the injection of a large amount of entropy in a second convergent process so that there is a conservative search and a somewhat wild search running in parallel.

It is true that quantum noise in extremely small analog circuits are have greater uniformity to the signal spectrum from its entropy than a digital pseudo-random generator and much fewer transistors are required to achieve the higher quality noise. Whether the challenges of doing so in VLSI implementations have been overcome is yet to be disclosed by research labs embedded in governments and corporations.

  • Will such stochastic elements used to inject measured quantities of randomness to enhance training speed and reliability be adequately immune to external noise during training?
  • Will they be sufficiently shielded from internal cross-talk?
  • Will a demand arise that will lower the cost of VLSI manufacture sufficiently to reach a point of greater use outside highly funded research enterprises?

All three challenges are plausible. What is certain and also very interesting is how designers and manufacturers facilitate digital control of the analog signal pathways and activation functions to achieve high speed training.

Footnotes

[1] https://ieeexplore.ieee.org/abstract/document/8401400/

[2] https://spectrum.ieee.org/automaton/robotics/artificial-intelligence/analog-and-neuromorphic-chips-will-rule-robotic-age

[3] https://www.roboticstomorrow.com/article/2018/04/whats-the-difference-between-analog-and-neuromorphic-chips-in-robots/11820

[4] Attenuation refers to multiplication of a signal output from one actuation by a trainable perameter to provide an addend to be summed with others for the input to an activation of a subsequent layer. Although this is a physics term, it is often used in electrical engineering and it is the appropriate term to describe the function of the vector-matrix multiplication that achieves what, in less educated circles, is called weighting the layer inputs.

[5] http://www.dtic.mil/dtic/tr/fulltext/u2/a256621.pdf

[6] There are many more than two parameters in artificial networks, but only two are depicted in this illustration because the plot can only be comprehensible in 3-D and we need one of the three dimensions for the error function value.

[7] Surface definition: $z = (x-2)^2 + (y-2)^2 + 60 - \frac {40} {\sqrt{1 + (y - 1.1)^2 + (x - 0.9)^2}} - \frac {40} {(1 + {((y - 2.2)^2 + (x - 3.1)^2)}^4)}$

[8] Associated gnuplot commands:

set title "Error Surface Showing How Global Optimum Can be Missed"
set xlabel "x"
set ylabel "y"
set pm3d at b
set ticslevel 0.8
set isosample 40,40
set xrange [0:4]
set yrange [0:4]
set nokey
splot (x-2)**2 + (y-2)**2 + 60 \
    - 40 / sqrt(1+(y-1.1)**2+(x-0.9)**2) \
    - 40 / (1+(y-2.2)**2+(x-3.1)**2)**4
$\endgroup$
4
$\begingroup$

Digital Instrumentation of the Analog Cells

One of the key challenges in analog artificial networks is that network instrumentation would be most practical if digital. Any VLSI implementation of analog perceptrons, convolutions, or spiking networks will likely need to have digital components in a hybrid arrangement for several functions.

  • Health indicators
  • Fault indicators
  • Archive and retrieval of learned parameters1
  • Overall system control
  • Setting hyper-parameters
  • Operational statistics
  • Introspection for development and debugging
  • Break points
  • Auditability

This means that the realization of a general purpose analog artificial learning network will require A-to-D and D-to-A conversion.2 The VLSI design challenge then becomes avoiding the build-up of transistors from the introduction of a large number of conversion blocks. Such would defeat the density advantage of the analog realization of forward and backward propagation.

The likely solution is to use a latching matrix to distribute signals from the D-to-A converters to capacitors and the low leakage switching matrix to select which value will be read by the A-to-D converters. This must be done without introducing digital noise into the analog paths and without degrading the stored charges or loss of accuracy in charging them.

How significant the number of additional transistors and routes in an out of the primary network circuit would be is can only be found by exercising a VLSI design process.

Important Open Source Contributions

The University of Massachusetts introduced the open source BindsNet repository3,4 in February 2018. It simulates analog spiking networks with digital software and hardware and leverages GPU acceleration through PyTorch.

This facilitates present day experimentation into spiking network designs and strategies. Success using simulation, if significant enough, would likely lead to a superior VLSI designs.


Footnotes

[1] In any practical learning system, learned parameters must be extracted from VLSI implementation, stored in a database, and made available to any number of development, test, UAT, or production systems for deployment, flaw root cause analysis, scaling, and disaster recovery. Saving and loading must be a basic feature of VLSI hybrid analog artificial networks, even between epochs during training and during actual field use.

[2] One cannot hold the learned state of an artificial network in capacitors indefinitely. Although capacitors have become the dominant passive component for analog circuits designed in standard CMOS processes, they cannot have much capacity and leakage is not zero. The half-life of the capacitive storage circuits and the required accuracy of parameter values will determine the rate of a read and conditional re-update cycle.

[3] BindsNet open source repository

[4] BindsNET [paper]: A machine learning-oriented spiking neural networks library in Python for the Harvard U publication of the abstract from the BindsNet paper.

$\endgroup$
4
$\begingroup$

I am surprised no one mentioned some of the specific research directions in the analog AI field. And also to clarify Artificial Intelligence is not exactly the same as Machine Learning as this answer suggests. Recent advances in analog computation has only been in the field of Machine Learning.

Analog CMOS:

First off let us talk about the earliest analog implementations of neurons. Dr.Giacomo Indiveri, et al has been few of the pioneers in the field. Although with CMOS logic you can design spiking Neural Nets with STDP (Spike Time Dependent Plasticity), it is difficult to make use of in Machine Learning algorithms. Human brain is yet to be fully understood, especially how it communicates complex information with spikes. The spike based networks are good in performing relatively small image recognition and low complexity tasks (Most papers seem to be more concerned about improving the performance rather than apply to highly complex tasks). Due to the sheer number of transistors available we might be able to make use of it in complex tasks.

The best example would be Google is using this idea of low precision in TPU's and compensating precision, by using huge number of processing units which is causing some kind of trade-off between time,precision and area. This can be analogous to huge number of transistors in a processor albeit with low precision. (An in-depth look at Google’s first Tensor Processing Unit (TPU))

NOTE: Some might argue CMOS technology falls under digital domain, but since we are not specifically using CMOS here to perform any digital operation I like to think of it as analog.

Spike based tasks are apparently quite good for Winner Take All networks (kind of like Self Organising Maps), so it is the general way of implementing Machine Learning Algorithms in VLSI chips.

Spike based networks do not have ideal memory, you cannot have high precision weights. They have proposed to implement biological weights or synapses or memory using capacitors, but apparently it faces problems similar to normal silicon chips, like charge leakage and also from other Silicon based non-idealities and from what I understood, they also can model limited weights (like -1, 0, 1).

Digital Computation:

Herein, comes digital computation. Tasks that require a high amount of floating point representation cannot simply be implemented by spikes, since we don't yet to know or even able to completely mimic the biophysical or any aspects of a true neuron for that matter. Digital computation simply helps in conveying more information as well with as much precision as we like (if we design such a CPU). Even though bottlenecks are a known drawback of Von Neumann architecture for digital computation it is not as much of a problem as information representation via spikes. Spikes always have a fixed magnitude, the only way it probably conveys information is by its frequency and sign (excitatory or inhibitory). Also clock speeds are pretty high in modern computers.

Memristors: A new direction

Herein comes the most recent invention, the Memristor. This by far has been the most promising analog device in Machine Learning. Memristors are a very new concept predicted in 70's and produced only in 2008.Basically, they are RRAM's or Resisitive RAM's. In this the resistance of the Memory Resistor or Memristor is directly related to the past current history which is very similar to biophysical models of neuron. They can also be trained easily using crossbar arrays (basically matrix of electrical contacts) of memristors (crossbar arrays will represent weight matrices, the voltage applied along rows or along columns determine forward propagation or backward propagation).

Thus Memristor gives a real analog spin to Machine Learning algorithms. Unfortunately, due to its recent arrival there are a lot of problems which are yet to be resolved.

  • Memristors can degrade quite quickly, that is they have limited training cycles.
  • Memristors introduce a lot of noise, which apparently does not help in the cause of regularisation as a ML engineer might think.
  • Exotic elements required to make it ($TiO_2$ and $HfO_2$) the users for Memristors in academic circles are very limited. But a few labs working on this area are:

Nano-electronics Research Laboratory, Purdue University

Electrochemical Materials, ETH Zurich

Human Brain Project

The MARCS Institute for Brain, Behaviour and Development

Neuromorphic Photonics:

Recently, there has been an interest in the field of Neuromorphic photonics. Here is a short article on the same. I am not familiar with the internal workings of the same, but AFAIK it involves the transmission of information in optical form within the processing chip itself. This leads to some of advantages over normal analog or digital circuits:

  • Faster information processing.
  • Higher information density.
  • Better data fidelity due to very less losses.
$\endgroup$
  • $\begingroup$ Side note: Some of my observations are fact based while some are purely from memory, so I might be wrong (since I am a beginner in this field). Feel free to point out mistakes. $\endgroup$ – DuttaA Sep 21 '18 at 18:19
2
$\begingroup$

I believe that most people have pretty much answered the question diligently in a really informative way. I would just like to say that we use digital circuits commonly because that is the existing technology and that definitely analog circuits seem really promising.

However, at this moment, this idea is not very well-developed despite the amount of research done in the past years. No company so far, has tried to implement the idea at a commercial level where they are making such chips for use outside their labs.

Besides, this idea feels like a new approach and has a great potential.

But, with our lack of understanding about how some models work, some just don't for a problem; how neural networks really solve such complex problems and many other things.Therefore, it is still quite a distant technology to reach its full potential.

PS I'm still a beginner in this field and think that my opinion does not count so, if I was redundant anywhere or failed to give you the expected answer then, I sincerely regret it.

$\endgroup$
  • $\begingroup$ This answer shows thought. It is true too that existing technology does not show as much progress with programmable analog VLSI as digital. ... What is unknown is the result of U.S. Navy and DARPA analog control R&D that has been amply funded for decades. Only initial documents have been declassified. ICBM and countermeasure technology may all be analog intelligence circuits in the 100 GHz range. Or not. ... Your writing was neither redundant nor naive. Surely, in the open source, these technologies are just starting to be seen. Good answer. Feel free to leave it as is or develop it further. $\endgroup$ – FauChristian Sep 17 '18 at 21:26
2
$\begingroup$

One can also approach the question from the information theory aspect:

There are two trade/offs to choose from:

Analog information that may represent information in a more precise/specific way, but limited in quantity.

Digital information that doesn't fully represent the real world, but may contain unlimited amount of information within a few bits. A good example could be something like a incrementing for loop:

i = 0
while True:
   print(i)
   i += 1

Which one is more powerful then?

$\endgroup$
  • $\begingroup$ That is generally true. Think about what that means to learning in the context of AI. We have simulated various kinds of learning in machines via rules systems with meta rules, artificial networks, extensions to Markov chain, fuzzy logic and a wide variety of other techniques and architectures. When learning occurs, there is some sort of optimal behavior that the learning attempts to acquire. How can analog or digital systems converge on or track (in real time) to that optimal behavior, and which has a long term advantage? $\endgroup$ – FauChristian Sep 18 '18 at 21:02
1
$\begingroup$

Hava Siegelmann

On the first look Analog computing is superior to digital one. Quantum computers are faster than Von-Neumann computers and neuromorphic chips need less energy than Intel CPUs. Also from a theoretic point of view many speaks for analog computers. Hava Siegelmann has researched the Super-turing capability of neural network, which means that an analog computer can emulate a digital one but not the other way around. So why should we not using analog computing?

Stephen Wolfram

The reason has to do with the education system. Classical mathematics which is teached in schools is analog mathematics. It is based on slide rules, logarithm table and the thinking in circuits. In contrast, thinking in discrete values of an algorithm and describe the world in zero and ones is fundamental different and leads us to a new kind of mathematics. Stephen Wolfram has explained, that the understanding of cellular automatons is an important step to describe the universe and he is right. Ignoring analog mathematics and preferring turing capable computer languages is a powerful method in education. It helps not only to become familiar with computers but with all the other things like medicine, literature and economy too. Even if analog machines are technical superior we should prefer slow but discrete Turing-machines, especially these one which implements AI-related algorithms.

Teaching mathematics

To understand the difference between digital and analog computation we must focus on the mathematics itself which is utilized in schools. If the idea is to push analog computation forward, the appropriate kind of mathematics is grouped around electrical fields, integration and differentiation. In schools this is teached under the umbrella term “Mathematical analysis”. This topic was very important in the past, because analysis helps to build bridges, machines and cars. In all of these domains vector algebra for describing geometric space is used.

If analog computation is so powerful, why does anybody need digital mathematics? It has to do with algorithm. What planimeter and differential analyzer doesn't have to offer are programming capabilities. It is not possible to define algorithms and artificial languages. A look into the history of mathematics shows, that algorithm-theory was not very common in the past. In modern mathematics it is discussed under the term Lambda calculus and Halting problem.

The funny thing is, that on the first look Lamda calculus has no practical applications. It is not needed if somebody want's to calculate the area of a bridge. Algorithm theory is a thought school to improve critical thinking. It is a philosophy needed by humans, not by machines.

$\endgroup$
  • $\begingroup$ Nice that you gave Seigelmann mention. The second paragraph is difficult to follow logically. Certainly education is central to this question, and DNA sequencing and digital imaging have definitely improved medicine. Can you elaborate on how literature has improved? Some would argue that digital computing has worsened the economy volatility, but more central to the bounty requirements, why anyone would prefer slow discrete over fast continuous doesn't follow from Wolfram's statement. There is no reference to the statement either. Can you provide a reference and provide the missing logic? $\endgroup$ – FauChristian Sep 17 '18 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.