2
$\begingroup$

In a neural network, a neuron typically computes a linear function $f(x) = w*x$, where $w$ is the weight and $x$ is the input.

Why not replacing the linear function with more complex functions, such as $f(x,w,a,b,c) = w*(x + b)^a + c$?

It will provide much more diversity into neural networks.

Does this have a name? Has this been used?

$\endgroup$
  • $\begingroup$ In your previous version of this post, you were using the word "weight" as if it was a function, but it's not. The weight is a number. The neuron computes some function. I think it's important you differentiate the two! So, it's the neuron that can compute more complex functions and "not the weight that can compute more complex functions". $\endgroup$ – nbro May 23 at 11:02
4
$\begingroup$

It is definitley possible to make the links between neurons use more complex functions. Provided those functions are differentiable, backpropagation still works, and the resulting compound function might be able to learn something useful. The general name for such a thing is a computational graph and the standardised structures used in most neural networks are a subset of all possible (and maybe useful) computational graphs.

When adding complex and non-linear functions into a neural network, this is usually alternated with simpler linear layers using the weights. A generalised function of a single neuron as used in most neural networks looks like this:

$$a = f(\sum_i w_i x_i + b)$$

Where $i$ indexes all inputs to the neuron, $x_i$ are the input values, $w_i$ the weights associated with each input, $b$ is a bias term and $f()$ is a differentiable non-linear activation function. Training process learns $w_i$ and $b$. The output $a$ is the the neuron's activation value, that may be taken as output of the neural network, or fed in to some other neuron as one of the next neuron's $x_i$.

A simple feed-forward network using this basic neuron function, with at least one hidden layer which has a non-linear activation function can already learn approximations to any given function - a result proved in the universal approximation theorem.

The practical result of the universal approximation theorem is that you need motivation other than increasing diversity in order to make neural network functions more complex. If you were considering altering one of the $w_i x_i$ multiplications, and replacing with a more complex learnable function, you can effectively achieve the same thing by adding another neuron whose output $a$ is used as $x_i$ - or simply adding a layer in most neural network libraries.

In some situations there may be good reasons to make lower-level changes:

  • If you know the function you are learning relates to a theoretical model with a specific mathematical form, you can deliberately set up functions that mirror that with learnable parameters. Typcially that is done as transforms on inputs, but could also be part of a more complex computational graph if necessary.

  • In neural network architectures, you can consider things such as gate combinations in LSTM cells, or skip connections in residual networks as examples where the functions have deliberately be made more complex to achieve a specific goal - in both those cases in order to increase effectiveness of backpropagation in deep structures.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.