Is there a way to understand neural networks without using the concept of brain?

Question

Is there a way to understand, for instance, a multi-layered perceptron without hand-waving about them being similar to brains, etc?

For example, it is obvious that what a perceptron does is approximating a function; there might be many other ways, given a labelled dataset, to find the separation of the input area into smaller areas that correspond to the labels; however, these ways would probably be computationally rather ineffective, which is why they cannot be practically used. However, it seems that the iterative approach of finding such areas of separation may give a huge speed-up in many cases; then, natural questions arise why this speed-up may be possible, how it happens and in which cases.

One could be sure that this question was investigated. If anyone could shed any light on the history of this question, I would be very grateful.

So, why are neural networks useful and what do they do? I mean, from the practical and mathematical standpoint, without relying on the concept of "brain" or "neurons" which can explain nothing at all.

Answer 1

tl;dr I always like to think of Neural Networks as a generalization of logistic regression.

I too don't like that, traditionally, when introducing Neural Networks, books start with biological neurons and synapses, etc. I think its more beneficial to start from statistics and linear regression, then logistic regression and then neural networks.

A perceptron is essentially a simple binary logistic regressor (if you threshold the output). If you have many perceptrons that share the same input (i.e. a layer in a neural network), you can think of it as a multi-class logistic regressor. Now, by stacking one such layer after an other, you create a Multi-Layer Perceptron (MLP), which is a Neural Network with two layers. There is equivalent to two multi-class logistic regressors stacked one after the other. One notable thing that changes is the training technique here, i.e. backpropagation (because you don't have direct access to the targets from the hidden layer). Another thing that can change is the activation function (it's not always sigmoid in Neural Networks)

Introduce sparse connectivity and weight sharing and you get a Convolutional Neural Network. Add a connection from a layer to its self (for the next timestep) and you get a Recurrent Neural Network. Likewise, you can reproduce any Neural Network through this reasoning.

I know this is an over-simplified way of presenting them, but I think you get the point.

Answer 2

One way to view a neural network is as a series of linear transformations.

You take a bunch of data points and look at it from a different perspective from a different space. You apply some non linear function on the data points like, ReLU, sigmoid etc. Now you repeat the same process of looking from a different space.

Our goal is to look at it from a point where things starts looking right for our tasks. These linear transformations is what the network has to optimise.

Answer 3

A good way of looking at it would be understanding neural networks mathematically, i.e. purely on the basis of the fact that you're just trying to fit a function and solve an optimisation problem (apart from looking at it as multiple units of logistic regression).

Say we want to approximate a function $y =f_w(x)$ with $x \in D$, where $D$ is our domain-space. We want this function to map to $C$, our co-domain, with all the values the function ends up taking being the set $y \in R$, our range. Essentially we frame $f(x)$ as a sequence of operations (What operation should be done where is got from common-practice, intuition, and insight mostly gained from experience) assuming that when the right parameters are used for these operations we will arrive at a very reasonable approximation of the function.

We initialise the parameters with whatever values we want initially (usually random), calling this parameter-space $W$. The essential idea would be frame another function $L(f_w(x), \hat{y})$ called the loss function which we want to minimise. This acts as a test to how good our function is - since our function parameters were initially random, the error between the function approximations and the actual range values for known points (training set) are estimated. These estimated error values and its gradient is then used by back-propagation where $w_{init}\in W$ is updated to another $w_{1}\in W$, where $w_1$ is calculated by moving on $L$ in the direction of decreasing gradient, in hopes of reaching the loss functions minima.

Simplifying, essentially all you want to do is find a $y=f_w(x)$ where parameters $w$ are to be chosen such that $L(f_w(x), \hat{y})$ is minimised for the training set.

Even though this is a very rough idea of neural networks, such a direction in thinking can especially be useful when studying generative networks and other problems where the problem has to be formulated mathematically before being able to approach it.