I'm certain that this is a very naive question, but I am just beginning to look more deeply at neural networks, having only used decision tree approaches in the past. Also, my formal mathematics training is more than 30 years in the past, so please be kind. :)

As I'm reading François Chollet's book on Deep Learning with Python, I'm struck that it appears that we are effectively treating the weights (kernel and biases) as terms in the standard linear equation ($y=mx+b$). At page 72 of the book, the author writes

output = dot(W, input) + b
output = (output < 0 ? 0 : output)

Am I reading too much into this, or is this correct (and so fundamental I shouldn't be asking about it)?


In a neural network (NN), a neuron can act as a linear operator, but it usually acts as a non-linear one. The usual equation of a neuron $i$ in layer $l$ of an NN is

$$o_i^l = \sigma(\mathbf{x}_i^l \cdot \mathbf{w}_i^l + b_i^l),$$

where $\sigma$ is a so-called activation function, which is usually a non-linearity, but it can also be the identity function, $\mathbf{x}_i^l$ and $\mathbf{w}_i^l$ are the vectors that respectively contain the inputs and the weights for neuron $i$ in layer $l$, and $b_i^l \in \mathbb{R}$ is a bias. Similarly, the output of a layer of a feed-forward neural network (FFNN) is computed as

$$\mathbf{o}^l = \sigma(\mathbf{X}^l \mathbf{W}^l + \mathbf{b}^l).$$

In your specific example, you set the new weight to $0$, if the output of the linear combination is less than $0$, else you use the output of the linear combination. This is the definition of the ReLU activation function, which is a non-linear function.

| improve this answer | |
  • $\begingroup$ Yes, I understand that... And I realize I was using ReLU rather than sigmoid... but am I wrong to see a clear relationship between how we are currently training networks and the classic function that defines a line? $\endgroup$ – David Hoelzer Oct 12 '19 at 16:27
  • $\begingroup$ Oh, and a follow-up... From my reading over the last week, I had the impression that sigmoid was out of favor in current thought. Is that not true? I realize it was the standard just a few years ago. $\endgroup$ – David Hoelzer Oct 12 '19 at 16:28
  • 1
    $\begingroup$ @DavidHoelzer In the equation of a line, $\sigma$ is always the identity function, while in the case of NNs, $\sigma$ is almost never the identity function. This is the main difference. $\endgroup$ – nbro Oct 12 '19 at 16:28
  • $\begingroup$ @DavidHoelzer It seems to me that ReLU (or variations) is used more often (than sigmoids), but I think you can find several discussions on the web related to this topic. $\endgroup$ – nbro Oct 12 '19 at 16:31
  • $\begingroup$ Don't you find the provided equation (by OP) strange? The output is supposedly the weight updation? $\endgroup$ – DuttaA Oct 13 '19 at 13:04

Almost never. The sum of linear functions is another linear function, so if neurons were only linear transformations there would be basically no point to having more than one neuron per layer. Instead, every neuron applies some kind of nonlinear function to its input. There are lots of different variations, but in the end the combination of the nonlinear activation function at each layer with the linear matrix multiplication connecting the outputs of each layer to the inputs of the next, creates something that has much more intricate behavior while still being reasonably efficient to compute.

| improve this answer | |
  • $\begingroup$ I know that the sum of a set of linear functions is linear... Perhaps I mislead you with my question. I was simply asking about the connection between how kernel and bias values are applied to produce weights; Obviously, we apply non-linear functions to the data during transformations. I appreciate your thoughts thought. $\endgroup$ – David Hoelzer Oct 13 '19 at 12:53

Taking the question from comments on nbro's answer.

Am I wrong to see a clear relationship between how we are currently training networks and the classic function that defines a line?

You are right about it. This is an intuitive way to understand neural networks. You can create a neural network that only does simple linear regression, by using linear activations functions in all the layers, such as the neural network (model) output is a linear combination of the inputs. And, this seems like a great way to introduce neural networks to students.

But, one must also look at the fact that neural networks provide the flexibility to model many kinds of non-linear relationships.

A list of activation functions.

| improve this answer | |
  • $\begingroup$ I very much appreciate your insight here. That's what I was driving at. $\endgroup$ – David Hoelzer Oct 14 '19 at 20:06
  • $\begingroup$ @DavidHoelzer I have added a list of activation functions, not exhaustive but covers many. Thank you. $\endgroup$ – naive Oct 15 '19 at 7:08

You're quite right in your interpretation, but I'll answer in two parts in order to avoid confusion with respect to activation functions.

Part 1. (TLDR: a neurons weights is the normal vector of a hyperplane that divides input space in two parts. The neuron's preactivation is proportional to the distance of the input point to the plane.) Every artificial neuron learns a linear relationship between its inputs. The most recalled equation of a line is $y=m \cdot x+b$, but that's actually a very specific form that allows us going through values of X of the line and seeing to what values of Y it corresponds. A most general form would be $0=n \cdot y + m \cdot x + b$. This tells us that the line is formed by the points (X,Y) whose values make that series of calculations be zero. We can explore different values of (X,Y) and see that most of them give non-zero values, and that they give positive values at one side of the line and negative values at the other side. Only if you land just on the line it will give you a zero. This is a very important interpretation, because it's what allows neurons to find divisions of the input space (into a positive side and a negative side). Of course it probably won't be a 2d space, so it will be a hyperplane instead of a line, but I hope you get the idea.

Part 2. However, if we only use linear transformations we couldn't learn non-linear functions. Here's where the activation function plays a very important role: it distorts the neuron's preactivation value (which is linear) in a non-linear way (what makes it a non-linear function). Activation functions have lots of bells and whistles, which are too much to write here, but you can start thinking about them as distortions applied to that distance of the input point to the neuron's hyperplane. The one you saw is called ReLU, and it basically truncates the negative values, thus focusing only on the positive side of the hyperplane (it may be interpreted as measuring how far the point has crossed a frontier).

| improve this answer | |
  • $\begingroup$ Wonderful. Thank you. $\endgroup$ – David Hoelzer Oct 15 '19 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.