What are some information processing models besides MLPs?

Question

Feedforward or multilayered neural networks, like the one in the image above, are usually characterized by the fact that all weighted connections can be represented as a continuous real number. Furthermore, each node in a layer is connected to every other node in the previous and successive layers.

Are there any other information processing models other than FFNNs or MLPs? For example, is there any system in which the topology of a neural network is variable? Or a system in which the connections between nodes are not real numbers?

Answer 1

Neural Network equivalents that is not (vanilla) feed forward Neural Nets:

Neural net structures such as Recurrent Neural Nets (RNNs) and Convolutional Neural Nets (CNNs), and different architectures within those are good examples.

Examples of different architectures within RNNs would would be: Long Short Term Memory (LSTM) or Gated Recurrent Unit (GRU). Both of these are well described in Colah's blog post on Understanding LSTMs

What are some alternative information processing system beside neural network

There are sooo many structures. From the top of my head: (Restricted) Boltzmann machine, auto encoders, monte carlo method and radial basis networks to name a few.

You can check out Goodfellow's Deep learning-book that is free online and get the gist of all the structures I mentioned here (most parts requires a bit of math knowledge, but he also writes about them quite intuitively).

For Recurrent Neural Nets I recommend Colah's blog post on Understanding LSTMs

Is there any system in which the topology of a neural network is variable?

Depends on what you mean with the topology of a neural network:

I think in the common meaning of topology when talking about Neural Networks is the way in which neurons are connected to form a network, varying in structure as it runs and learns. If this is what you men then the answer, in short, is yes. In multiple ways actually. On the other hand, if you mean in the mathematical sense, this answer would become a book that I wouldn't feel confortable writing. So I'll assume you mean the first.

We often do "regularization", both on vanilla NN and other structures. One of these regularization techniques are called dropout, which would randomly remove connections from the network as it is training (to prevent something called overfitting, which I'm not gonna go into in this post).

Another example for another way would be on the Recurrent Neural Network. They deal with time series, and are equipped for dealing with timeseries of different lengths (thus, "varying structure").

Does it exist neural net systems where complex numbers are used?

Yes, there are many papers on complex number machine learning structures. A quick google should give you loads of results. For example: DeepMind has a paper on Associative Long Short-Term Memory which explores the use of complex values for an "associative memory".

Links:

Goodfellow's Deep Learning-book: deeplearningbook.org

Colah's blogpost on RNN's: colah.github.io

Paper on DeepMinds Associative LSTM: arxiv:1602.03032

Answer 2

To answer the title, there are many other machine learning models, but neural networks work particularly well for some difficult problems (image classification, speech recognition) which is one of the reasons they have gained popularity.

Two particularly simple models are the decision tree and the perceptron. These are rather simple models, but they both have redeemable qualities. A decision tree is useful as it provides a model that is easily understood, while a perceptron is fairly quick and works well for linearly separable data. Another, more advanced, model is the Support Vector Machine.

For example, is there any system in which the topology of a neural network is variable?

Yes, there are many such systems where the topology of the neural network is dynamic throughout training. An entire class of methods labeled TWEANNs are designed to evolve the topology of the networks, one such algorithm is NeuroEvolution of Augmenting Topologies, NEAT (and it's descendants rtNEAT, hyperNEAT, ...).

Answer 3

A very popular choice are Hidden Markov Models.