7
$\begingroup$

The idea that weights determine how much influence each input value from the current layer will have when calculating the input to the following layer reminds me of when my professors would say that our final grade for the course was broken down something like this:

Item Percent of Grade
Midterm Exam 1 20%
Midterm Exam 2 20%
Final Exam 50%
Classwork 5%
Homework 5%
Total 100%

My question is then do all the weights need to add up to 1.0 (i.e. 100%)? If not, why?

$\endgroup$
6
  • $\begingroup$ Why "If not, why?" and not "If yes, why?" $\endgroup$
    – Stef
    Sep 4, 2023 at 17:10
  • $\begingroup$ @Stef I think I just assumed both were implied when I was thinking of the question, didn't think much of it, but valid point $\endgroup$
    – Garrett
    Sep 4, 2023 at 19:33
  • $\begingroup$ ;) You'll notice one of the answers begins with "No, the weights do not need to add up to one. There isn't really a reason to do that." and another with "No. Why should they?". There are situations where imposing the constraint "must sum to 1" is useful, but the weights of a neural network is not such a situation. Since it's not needed, we don't impose it. Scolpe's answer gives one argument why imposing it would be a bad idea. One place where this constraint would be useful is in the outputs of a classification neural net, where the outputs do represent probabilities (but not the weights) $\endgroup$
    – Stef
    Sep 4, 2023 at 19:45
  • $\begingroup$ I don't understand what the relationship between the grades and the weights you thought could possibly exist. It seems that something is missing from this question. Some people in the answers even mention probability distributions, which you don't mention. This seems like a random question. Maybe a question that has not been asked here, or maybe yes. Still, the motivation for this question is unclear to me. It's unclear where the doubt is. It's unclear why you're asking this question. Still, it makes us think why. So, why not challenge the status quo? $\endgroup$
    – nbro
    Sep 5, 2023 at 9:24
  • $\begingroup$ @nbro I didn't originally think of weights in this way. I'm a visual learner and trying to think of real-world things I could compare weights to to help me visualize them and this idea came up. $\endgroup$
    – Garrett
    Sep 5, 2023 at 11:11

4 Answers 4

8
$\begingroup$

No, the weights do not need to add up to one. There isn't really a reason to do that. Weights as "contributions" may not be the best way to think about things here -- you're trying to learn a function that transforms the input values into your desired output values. Say you're learning a linear model and your data is 2D and drawn from a line with slope 2, then you'd want your learned weight to be 2.

Additionally, say you have a feature that is negatively correlated with the output. That is, that a higher value in the input results in a lower outputs value. If all your weights are non-negative, you wouldn't be able to encode this at all!

Finally, as the output would simply be a weighted average of the inputs, the scale of the outputs would be constrained by the scale of the inputs. Consider: $$f(\mathbf{x}) = \Sigma^d_{i=1} w_i x_i$$ with $$\Sigma^d_{i=1} w_i = 1$$

$f(\mathbf{x})$ will always be less than or equal to $\max(\{w_1, ..., w_d\})$. For a simple linear model, you can definitely think of examples where the scale of the output is much larger than that of the input features (for example, number of bathrooms in a house versus its price). Although you can mitigate some of this by rescaling your data (as I will talk about shortly), there is really no reason to artificially constrain your model like this.

A deep neural network can be thought of as stacks of these linear models, with the output of the previous layer being the input of the next layer. Although, these intuitions aren't at all perfectly transferable to deep neural networks, we would still be constraining the expressiveness of the model in a similar way: an increase in an input dimension can't result in a decrease in the output value and the scale of the output is constrained by the scale of the input.

Normalization in Neural Networks

You do, however, often need to perform normalization, but this normalization is done on the data and the outputs of hidden layers instead of the weights. If you add a weight penalty to your linear model for example, you want to make sure that each dimension is in around the same range, to not overly penalize features that tend to have larger weights. In neural networks, you also often normalize weight values for hidden layers, the most common variant is Batch Normalization.

However, the normalization often used in these cases is a bit different from what you mentioned. Also known as standardization, this kind of normalization subtracts each sample by the mean (over a set of samples) and divides by the standard deviation (over that same set of samples).

The exact reason why this is effective in neural networks is an active area of research. A common explanation is batch normalization reduces the effects of internal covariant shift, where changes in weights during training results in changes in the output distribution of a layer, which can harm performance as the next layer "expects" the original distribution. Centering and scaling the input to have zero mean and unit standard deviation reduces these changes. This, however, is disputed by this paper, which argues that batch normalization helps by making the loss landscape smoother.

$\endgroup$
6
$\begingroup$

Nothing to do with it! Let me illustrate this concept using one of the simplest models in machine learning: Linear Regression. In linear regression, the goal is to find the coefficients of a hyperplane that best fits the given data, represented by the equation $Ax=b$. Here, we have free variables that represent the slopes along each axis and an independent term for the height (that in neural nets is called the Bias). Now, to address your question: Do the coefficients need to add up to 1? The answer is no.

You might be tempted to interpret the concept of weights with the concept of probabilities, but they are not the same! They are merely coefficients in a high-dimensional, nested function.

enter image description here

Here's a 3D example of linear regression. The blue points represent the data, and the red plane is the best-fitting hyperplane according to the linear regression model.

The equation of the fitted plane is approximately:

$y=2.79⋅x_1+2.01⋅x_2+4.11$

As you can see, the coefficients are $2.79$ for $x_1$ $2.01$ for $x_2$ , and the intercept is $4.11$. If we sum these together, we get $2.79 + 2.01 + 4.11 = 8.91$, which is clearly not equal to $1$.

$\endgroup$
3
  • $\begingroup$ Couldn't you simply reparameterise the coefficients to [0.2817, 0.2026, 0.4157] and have them add up to 1 again? $\endgroup$ Sep 6, 2023 at 9:09
  • $\begingroup$ I think the main point here is the term "weights" has nothing to do with the term "weights" which represents a percentage. $\endgroup$
    – osbm
    Sep 6, 2023 at 14:27
  • 1
    $\begingroup$ @ToivoSäwén If we parameterize the coefficients to the values you mentioned in order for them to add up to one, we will not end up with the same hyperplane. In the example I gave, we had the following linear equation: y = 2.79x₁ + 2.01x₂ + 4.11 If we divide the entire equation by 8.91, we get: y' = 0.31x₁ + 0.225x₂ + 0.416 This changes the y value to y'. Therefore, we would have to change our data "y" value in order for it to work. I believe you are confusing normalizing a vector with the property that all its components add up to one. $\endgroup$
    – Cesar Ruiz
    Sep 6, 2023 at 20:36
3
$\begingroup$

do all the weights need to add up to 1.0 (i.e. 100%)?

No.

If not, why?

Why should they?

In the case of school grades, the justification might be that each component (midterm exam, homework etc.) is graded separately on a same scale and the final grade, which must lie on the same scale, is computed as the weighted average of these components.

There is no such requirement regarding weights in neural networks.

What you might want to enforce is that the outputs in the last layer sum to one, to obtain probabilities of class memberships (this, obviously, applies only to classification problems and not to regression). Softmax is typically used for that purpose.

$\endgroup$
3
$\begingroup$

No, weights in that case do not constitute a formal probability distribution and do not need to be add up to 1. In fact, if they would — it could cause problems with underflow. Imagine a (nonsensical) neural network — n inputs, followed by 10 layers, each containing 10k parameters. This MLP would possess 100,000 parameters, excluding biases and the output layer. Given the novel architectures, it is not even a large number of parameters. Now, we normalize all the weights (layer or network-wise) so they form a valid probability distribution. It is easy to see, that the forward and backward passes could be problematic as the value of each parameter would be close to 0. Also, each gradient could be negligible small - a phenomenon that is observable now in NN due to various reasons Vanishing Gradient Problem.

How do we interpret the activity of the Neural Networks? It is a reasonable question — there is a field of research that is called an Explainable Artificial Intelligence and deals with the interpretation of many deep learning architectures. Given that deep learning basically provides for creating a sequence of nested functions that can approximate arbitrary difficult non-linear functions it is a non-trivial task. If the weights would sum up to 1 — it could greatly help the progress. Unfortunately, this is not the case ;)

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .