Are there names for neural networks with a well-defined layer or neuron characteristics?

Question

Are there names for neural networks with a well-defined layer or neuron characteristics?

For example, a matrix that has the same number of rows and columns is called a square matrix.

Is there an equivalent for classifying different neural network structures. Specifically, I am interested if there is a name for a neural network with x number of layers, but each layer has the same number of neurons?

Answer 1

Neural networks (NNs) are usually classified into feed-forward (i.e. NNs with feedforward connections), recurrent (i.e. NNs with recurrent connections) and convolutional (i.e. NNs that perform a convolution or cross-correlation operation). The term multi-layer NN may also be used to refer to feed-forward neural networks or, in general, neural networks with multiple (hidden) layers. There are also the perceptrons, which do not have hidden layers (i.e. the inputs are directly connected to the outputs).

You may still divide neural networks into classifiers (i.e. the outputs and labels are discrete) and regressors (the outputs and labels are numerical). Furthermore, you may classify them into generative models (e.g. the VAE) or discriminative models.

By analogy with linear algebra concepts, each layer of a feedforward neural network (FFNN) can be seen as a linear operation followed by an element-wise application of a linear or non-linear activation function.

$$ \mathbf{o}^{l} = \sigma \left(\mathbf{a}^{l} \mathbf{W}^{l} + \mathbf{b}^{l}\right) $$

where $\sigma$ is the activation function, $\mathbf{a}^{l}$ the inputs to the layer $l$ and $\mathbf{W}^{l}$ the parameters of the layer $l$ (similar to the parameters or coefficients in linear regression) and $\mathbf{b}^{l}$ is the bias vector (a scalar bias for each neuron) of layer $l$. $\mathbf{o}^{l}$ will then be $\mathbf{a}^{l+1}$ (i.e. the input to the next layer).

A recurrent neural network (RNN) performs a slightly more complex operation.

$$ \mathbf{o}^{l}_t = \sigma \left(\mathbf{a}^{l}_t \mathbf{W}^{l} + \mathbf{o}^{l}_{t-1} \mathbf{R}^{l} + \mathbf{b}^{l}\right) $$

where, in addition to the matrix $\mathbf{W}^{l}$ associated with the feedforward connections, it also uses another matrix $\mathbf{R}^{l}$ associated with the recurrent connections (i.e. cyclic or loopy connections of the neurons), which is multiplied by the output of the same layer but at the previous time step. $\mathbf{o}^{l}_t$ may actually just be the state of the recurrent layer, which is then used to compute the actual output of the layer, but, for simplicity, you can ignore this. Furthermore, note that there are more complex recurrent architectures, but this is the basic idea.

A convolutional neural network (CNN) performs the convolution (or cross-correlation) operation. If you are familiar with signal processing, e.g. kernels, convolution, etc., then you can view a CNN as performing a convolution (or cross-correlation) operation. It's actually possible to view the convolution as a matrix multiplication operation, but the details can easily become cumbersome and tedious to explain in an answer. A CNN may also perform other types of operations (such as downsampling) and it can also be composed of a feedforward part (usually, the last layers of a CNN are feedforward layers), but a CNN is a CNN because it performs the convolution (or cross-correlation).

In all cases, the matrices do not necessarily take any particular form (e.g. they are not necessarily square matrices). The dimensionality of the matrices depends on the number of layers and connections in the network, which can vary depending on many factors (e.g. the need for recurrent connections because they are useful for sequence modeling). These matrices (along with the biases) are the learnable parameters of the networks, but the learned values of these matrices highly depend on your data, the way you initialize them before learning, the architecture of the network, the learning algorithm, etc.

In the case of the FFNN, if the previous layer $l-1$ has the same number of neurons as the current layer $l$, then $\mathbf{W}^{l}$ is a square matrix. AFAIK, there is no name for a neural network with the same number of neurons for each layer. It could be called a rectangular neural network (but this is a name I've just come up with).

To conclude, there are many different neural networks and taxonomies for neural networks, so it is impossible to list or discuss them all in an answer, but, nowadays, the subdivision into feedforward, recurrent and convolution is the most common and general. See e.g. the paper A Taxonomy for Neural Memory Networks if you are interested in a more detailed taxonomy for memory-based neural networks (e.g. recurrent neural networks).