12
$\begingroup$

Is there a way to understand, for instance, a multi-layered perceptron without hand-waving about them being similar to brains etc?

For example: it is obvious that what a perceptron does is approximating a function; there might be many other ways, given a labelled dataset, to find the separation of the input area into smaller areas that correspond to the labels; however, these ways would probably be computationally rather ineffective, which is why they cannot be practically used. However, it seems that the iterative approach of finding such areas of separation may give a huge speed-up in many cases; then, natural questions arise why this speed-up may be possible, how it happens and in which cases.

One could be sure that this question was investigated. If anyone could shed any light on the history of this question, I would be very grateful. So, why are neural networks useful and what do they do? I mean, from the practical and mathematical standpoint, without relying on the concept of "brain" or "neurons" which can explain nothing at all.

$\endgroup$
  • 1
    $\begingroup$ I haven't seen any explanation of NNs relying on brain except just for an initial motivation of NN structure in earlier days. As far as separation of data is concerned it depends on the problem, lower dimensional data can easily be separated by definite statistical methods but its time complexity increases drastically for higher dimensional data, hence the use of NNs. Also clustering Algorithms although unsupervised are much faster than NNs. My way of viewing NNs are as something analogous to Fourier Series (a sum of sinusoidal functions with a tunable parameter to approximate a periodic func) $\endgroup$ – DuttaA Oct 19 at 18:39
  • $\begingroup$ @DuttaA Yes, thank you, that is exactly what (part of) my question is about. In fact, I assume that I may not understand the answers, but I still think this question has its place here. I wondered whether the structure of NNs could be motivated differently, by different ideas, I mean if we forget all about brains and just have a problem at hand, either to approximate a function or to label data clusters. $\endgroup$ – Evgeniy Oct 19 at 18:51
  • $\begingroup$ It's the result of highly specialized research in the 20th century that research groups are working by it's own and the vocabulary is different. Neurophysiologists are focussed on neural networks, while computer scientists are using frames and scripts to explore the subject of symbolic AI. $\endgroup$ – Manuel Rodriguez Oct 19 at 19:41
  • $\begingroup$ Excellent question. Thanks for contributing, and hope to see more of you on SE:AI! $\endgroup$ – DukeZhou Nov 14 at 22:50
9
$\begingroup$

tl;dr I always like to think of Neural Networks as a generalization of logistic regression.

I too don't like that, traditionally, when introducing Neural Networks, books start with biological neurons and synapses, etc. I think its more beneficial to start from statistics and linear regression, then logistic regression and then neural networks.

A perceptron is essentially a simple binary logistic regressor (if you threshold the output). If you have many perceptrons that share the same input (i.e. a layer in a neural network), you can think of it as a multi-class logistic regressor. Now, by stacking one such layer after an other, you create a Multi-Layer Perceptron (MLP), which is a Neural Network with two layers. There is equivalent to two multi-class logistic regressors stacked one after the other. One notable thing that changes is the training technique here, i.e. backpropagation (because you don't have direct access to the targets from the hidden layer). Another thing that can change is the activation function (it's not always sigmoid in Neural Networks)

Introduce sparse connectivity and weight sharing and you get a Convolutional Neural Network. Add a connection from a layer to its self (for the next timestep) and you get a Recurrent Neural Network. Likewise, you can reproduce any Neural Network through this reasoning.

I know this is an over-simplified way of presenting them, but I think you get the point.

$\endgroup$
  • $\begingroup$ Thank you, looks like that's a way to go, also looks like I have a lot to learn. :) $\endgroup$ – Evgeniy Oct 24 at 13:04
2
$\begingroup$

One way to view a neural network is as a series of linear transformations.

You take a bunch of data points and look at it from a different perspective from a different space. You apply some non linear function on the data points like, ReLU, sigmoid etc. Now you repeat the same process of looking from a different space.

Our goal is to look at it from a point where things starts looking right for our tasks. These linear transformations is what the network has to optimise.

$\endgroup$
  • $\begingroup$ That is to say, one neuron in a layer makes one coordinate, and the resulting points are then sharply differentiated by a non-linear function. So, approximation will be almost linear at some areas that are not covered by the network's "memory", and quite non-linear at other areas where the "lines of separation" are drawn and fixed in the network's weights. As the points on the second layer correspond to whole areas on the first layer, the resulting "lines of separation" may take on all kinds of shapes, not just a direct line as on the first layer.. $\endgroup$ – Evgeniy Oct 30 at 7:53
0
$\begingroup$

A good way of looking at it would be understanding neural networks mathematically, i.e. purely on the basis of the fact that you're just trying to fit a function and solve an optimisation problem (apart from looking at it as multiple units of logistic regression).

Say we want to approximate a function $y =f_w(x)$ with $x \in D$, where $D$ is our domain-space. We want this function to map to $C$, our co-domain, with all the values the function ends up taking being the set $y \in R$, our range. Essentially we frame $f(x)$ as a sequence of operations (What operation should be done where is got from common-practice, intuition, and insight mostly gained from experience) assuming that when the right parameters are used for these operations we will arrive at a very reasonable approximation of the function.

We initialise the parameters with whatever values we want initially (usually random), calling this parameter-space $W$. The essential idea would be frame another function $L(f_w(x), \hat{y})$ called the loss function which we want to minimise. This acts as a test to how good our function is - since our function parameters were initially random, the error between the function approximations and the actual range values for known points (training set) are estimated. These estimated error values and its gradient is then used by back-propagation where $w_{init}\in W$ is updated to another $w_{1}\in W$, where $w_1$ is calculated by moving on $L$ in the direction of decreasing gradient, in hopes of reaching the loss functions minima.

Simplifying, essentially all you want to do is find a $y=f_w(x)$ where parameters $w$ are to be chosen such that $L(f_w(x), \hat{y})$ is minimised for the training set.

Even though this is a very rough idea of neural networks, such a direction in thinking can especially be useful when studying generative networks and other problems where the problem has to be formulated mathematically before being able to approach it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.