What are the models that have the potential to replace neural networks in the near future?

Question

Are there possible models that have the potential to replace neural networks in the near future?

And do we even need that? What is the worst thing about using neural networks in terms of efficiency?

Answer 1

This is going backwards, but it kind of follows the logic of the arguments.

In terms of efficiency, I can see a few major problems with classical neural networks.

Data collection and preprocessing overhead

Large neural networks require a lot of data to train. The amount can vary depending on the size of the network and the complexity of the task, but as a rule of thumb it is usually proportional to the number of weights. For some supervised learning tasks, there simply isn't enough high-quality labelled data. Collecting large amounts of specialised training data can take months or even years, and labelling can be cumbersome and unreliable. This can be partially mitigated by data augmentation, which means "synthesising" more examples from the ones you already have, but it is not a panacea.

Training time vs. energy tradeoff

The learning rate is usually pretty small, so the training progress is slow. A large model that could take weeks to train on a desktop CPU can be trained in, say, two hours by using a GPU cluster which consumes several kW of power. This is a fundamental tradeoff due to the nature of the training procedure. That said, GPUs are getting increasingly efficient - for example, the new nVidia Volta GPU architecture allows for 15.7 TFLOPs while consuming less than 300 W.

Non-transferrability

Right now, virtually every different problem requires a custom neural network to be designed, trained and deployed. While the solution often works, it is kind of locked into that problem. For example, AlphaGo is brilliant at Go, but it would be hopeless at driving a car or providing music recommendations - it was just not designed for such tasks. This overwhelming redundancy is a major drawback of neural networks in my view, and it is also a major impediment to the progress of neural network research in general. There is a whole research area called transfer learning which deals with finding ways of applying a network trained on one task to a different task. Often this relates to the fact that there might not be enough data to train a network from scratch on the second task, so being able to use a pre-trained model with some extra tuning is very appealing.

---

The first part of the question is more tricky. Leaving purely statistical models aside, I haven't seen any prominent approaches to machine learning that are radically different from neural networks. However, there are some interesting developments that are worth mentioning because they address some of the above inefficiencies.

Neuromorphic chips

A bit of background first.

Spiking neural networks have enormous potential in terms of computational power. In fact, it has been proven that they are strictly more powerful than classical neural networks with sigmoid activations.

Added to that, spiking neural networks have an intrinsic grasp of time - something that has been a major hurdle for classical networks since their inception. Not only that, but spiking networks are event-driven, which means that neurons operate only if there is an incoming signal. This is in contrast to classical networks, where each neuron is evaluated regardless of its input (again, this is just a consequence of the evaluation procedure usually being implemented as a multiplication of two dense matrices). So spiking networks employ a sparse encoding scheme, which means that only a small fraction of the neurons are active at any given time.

Now, the sparse spike-based encoding and event-driven operation are suitable for hardware-based implementations of spiking networks called neuromorphic chips. For example, IBM's TrueNorth chip can simulate 1 million neurons and 256 million connections while drawing only about 100 mW of power on average. This is orders of magnitude more efficient than the current nVidia GPUs. Neuromorphic chips may be the solution the training time / energy tradeoff I mentioned above.

Also, memristors are a relatively new but very promising development. Basically, a memristor is a fundamental circuit element very similar to a resistor but with variable resistance proportional to the total amount of current that has passed through it over its entire lifetime. Essentially, this means that it maintains a "memory" of the amount of current that has passed through it. One of the exciting potential applications of memristors is modelling synapses in hardware extremely efficiently.

Reinforcement learning and evolution

I think these are worth mentioning because they are promising candidates for addressing the problem of non-transferrability. These are not restricted to neural networks - being reward-driven, RL and evolution are theoretically applicable in a generic setting to any task where it is possible to define a reward or a goal for an agent to attain. This is not necessarily trivial to do, but it is much more generic than the usual error-driven approach, where the learning agent tries to minimise the difference between its output and a ground truth. The main point here is about transfer learning: ideally, applying a trained agent to a different task should be as simple as changing the goal or reward (they are not quite at that level yet, though...).

Answer 2

We do have some hope lurking in that front. As of now we have capsule networks by J.Hinton which uses a different non-linear activation called the 'squash' function.

1. Hinton calls max-pooling in CNN as a 'big mistake', as CNN look only for presence objects in an image rather than the relative orientation between them. So they lose the spatial information while trying to achieve translation invariance.
2. Neural nets have fixed connections, whereas a capsule in a capsule network 'decides' to which other capsule it has to pass its activation during every epoch. This is called 'routing'.
3. The activation of every neuron in neural nets is a scalar. Whereas the activation of capsule is a vector capturing the pose and orientation of an object in an image.
4. CNN are considered to bad representations of human visual system. By human visual system I mean eyes and the brain/cognition together. We could identify Statue of Liberty from any pose, even if we have looked at it from one pose. CNN on most of the cases cannot detect same object in different poses and orientations.

Capsule networks themselves have some shortcomings. So there has been work in the direction of looking beyond neural nets. You can read this blog for a good understanding before you read the paper by J.Hinton.

Answer 3

Replacing Neural Nets

There may exist new algorithms that have the potential to replace neural nets. However, one of the characteristics of neural nets is that they employ simple elements, each with low demands on computing resources in geometric patterns.

Artificial neurons can be run in parallel (without CPU time sharing or looping) by mapping the computations to DSP devices or other parallel computing hardware. That the many neurons are essentially alike is thus a strong advantage.

What Would We Be Replacing?

When we consider algorithmic replacements to neural nets, we imply that a neural net design is an algorithm. It is not.

A neural net is an approach to converging on a real time circuit to perform a nonlinear transformation of input to output based on some formulation of what is optimal. Such a formulation may be the minimization of a measure of error or disparity from some defined ideal. It may be a measure of wellness that must be maximized.

The source of the fitness determination for any given network behavior may be internal. We call that unsupervised learning. It may be external, which we call supervised when the external fitness information is coupled with input vectors in the form of desired output values, which we call labels.

Fitness may also originate externally as a scalar or vector not coupled with the input data but rather real time, which we call reinforcement. Such requires re-entrant learning algorithms. Net behavioral fitness may alternatively be evaluated by other nets within the system, in the case of stacked nets or other configurations such as Laplacian hierarchies.

The selection of algorithms has little to do with comparative intelligence once the mathematical and process designs are selected. Algorithm design is more directly related to minimizing demands for computing resources and reducing time requirements. This minimization is hardware and operating system dependent too.

Is a Replacement Indicated?

Sure. It would be better if networks were more like mammalian neurons.

• Sophistication of activation
• Heterogeneity of connection patterns
• Plasticity of design, to support meta-adaptation
• Governed by many dimensions of regional signaling

By regional signaling is meant the many chemical signals beyond signal transmission across synapses.

We can even consider going beyond mammalian neurology.

• Combining parametric and hypothesis-based learning
• Learning of the form employed when microbes pass DNA

Neural Net Efficiency

Efficiency cannot be quantified in some universal scale as temperature can be quantified in degrees Kelvin. Efficiency can only be quantified as a quotient of some measured value over some theoretical ideal. Note that it is an ideal, not a maximum, in the denominator. In thermodynamic engines, that ideal is the rate of energy input, which can never be fully transferred to the output.

Similarly, neural nets can never learn in zero time. A neural net cannot achieve zero error over an arbitrarily long time in production either. Therefore information is in some ways like energy, a concept investigated by Claude Shannon of Bell Labs during the dawn of digital automation, and the relationship between information entropy and thermodynamic entropy is now an important part of theoretical physics.

There can be no bad learning efficiency or good learning efficiency. There can be neither bad performance nor good performance, if we wish to think in logical and scientific terms — only relative improvement of some system configuration with respect to some other system configuration for a very specific set of performance scenarios.

Therefore, without an unambiguous specification of the two hardware, operating system, and software configurations and a fully defined test suite used for relative evaluation, efficiency is meaningless.

Answer 4

Neural networks require lots of data and training. For most tabular format datasets it is much better to use decision tree based models. Most of the time, simple models are enough to give good accuracy. However neural networks had their test of time. It has only been five to six years since the deep learning revolution started, so we still do not know the true potency of deep learning.