Skip to main content
27 votes

In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the 3D filter (grey cube) is made up of a different value (...
Mohsin's user avatar
  • 982
16 votes

In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

In a convolutional neural network, is there a unique filter for each input channel or are the same new filters used across all input channels? The former. In fact there is a separate kernel defined ...
Neil Slater's user avatar
  • 32.7k
13 votes
Accepted

Why are the initial weights of neural networks randomly initialised?

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue. http://www.deeplearningbook.org/contents/optimization.html Perhaps the only property known with complete ...
SmallChess's user avatar
  • 1,411
8 votes
Accepted

Do we know what the units of neural networks will do before we train them?

In reverse order to how you asked: all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal This actually happens if you ...
Neil Slater's user avatar
  • 32.7k
8 votes

In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

I'm following up on the answers above with a concrete example in the hope to further clarify how the convolution works with respect to the input and output channels and the weights, respectively: Let ...
Lukas Z.'s user avatar
  • 209
8 votes

Do neural network weights need to add up to one?

No, the weights do not need to add up to one. There isn't really a reason to do that. Weights as "contributions" may not be the best way to think about things here -- you're trying to learn ...
Alexander Wan's user avatar
7 votes

Why are the initial weights of neural networks randomly initialised?

The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to ...
Aiden Grossman's user avatar
6 votes

Do neural network weights need to add up to one?

Nothing to do with it! Let me illustrate this concept using one of the simplest models in machine learning: Linear Regression. In linear regression, the goal is to find the coefficients of a ...
Cesar Ruiz's user avatar
5 votes
Accepted

What is the significance of weights in a feedforward neural network?

You described a single-layer feedforward network. They can have multiple layers. The significance of the weights is that they make a linear transformation from the output of the previous layer and ...
Didami's user avatar
  • 391
5 votes

Why are the initial weights of neural networks randomly initialised?

That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, Gradient Descent Finds Global Minima of ...
mirror2image's user avatar
4 votes
Accepted

Can neurons in MLP and filters in CNN be compared?

tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer Differences The neurons of these two types of layers have two key differences. These ...
Djib2011's user avatar
  • 3,193
4 votes
Accepted

What is the goal of weight initialization in neural networks?

Is it trying to make sure there is no symmetry in the gradients? The aim of weight initialization is to make sure that we don't converge to a trivial solution. That's why we have different kinds of ...
Saurav Maheshkar's user avatar
4 votes
Accepted

Is the bias also a "weight" in a neural network?

Yes, it is not unusual to omit the bias by adding a neuron which always outputs a constant 1, which will then be multiplied by an appropriate weight to give the same formula as you would get using an ...
htl's user avatar
  • 1,010
4 votes

Can some of the weights be fixed during the training of a neural network?

Yes, you can fix (or freeze) some of the weights during the training of a neural network. In fact, this is done in the most common form of transfer learning (which is described here). I don't know ...
nbro's user avatar
  • 40.9k
4 votes

Is there a proper initialization technique for the weight matrices in multi-head attention?

IMO xavier/glorot is the correct way to initialize the $W_Q$ and $W_K$ matrices. In section 3.2.1 of the transformer paper the ...
pi-tau's user avatar
  • 815
3 votes
Accepted

How are non-linear surfaces formed in the training of a neural network?

Hi and welcome to the community. It's important to understand these basic concepts very clearly. You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let ...
Ananda's user avatar
  • 148
3 votes

How are newer weight initialization techniques better than zero or random initialization?

There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, Xavier et al.. ...
hola's user avatar
  • 381
3 votes

How does the neural-network know how to tweak weights for a specific neuron?

tl;dr The whole point of gradient descent is to assess the contribution of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter. A ...
Djib2011's user avatar
  • 3,193
3 votes
Accepted

Neural Nets: CNN confirming layer/filter arithmetic

Your first point is correct. The filters are stored in 4d arrays, with dimensions of (height, width, input channels, filter number) . The order may differ. Your second point is correct too. The ...
Clement's user avatar
  • 1,745
3 votes

In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

I recommend chapter 2.2.1 of my masters thesis as an answer. To add to the remaining answers: Keras is your friend to understand what happens: ...
Martin Thoma's user avatar
  • 1,055
3 votes
Accepted

Which hyperparameters in neural network are accesible to users adjustment

In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are: learning rate, ...
ddaedalus's user avatar
  • 919
3 votes
Accepted

Not able to understand Pytorch Tensor (Weight & Biases) Size for Linear Regression

The size of the parameters tensor is depended on what type of layer that you want to build. Convolutional, fully connected, attention or even custom layer, each layer has a difference in the way it ...
CuCaRot's user avatar
  • 912
3 votes
Accepted

Why and when do we need to normalize weights in Reinforcement Learning?

The kind of divergence that the other question experienced is a common problem with deep RL and temporal difference methods (Q-learning, SARSA, or any Actor Critic). The weight normalisation would not ...
Neil Slater's user avatar
  • 32.7k
3 votes

What is actually being saved in the file when you save a model? For example a Tensorflow SavedModel file

Answer to Question 1 TensorFlow's documentation provides the following information on what is saved: The model config, weights, and optimizer are included in the SavedModel. Additionally, for every ...
Brian O'Donnell's user avatar
3 votes
Accepted

Why does averaging attention-weighted positions reduce the effective resolution in transformers?

I think that the idea of "reduced effective resolution" from averaging is best understood through seeing how the proposed multi-head attention architecture fixes the issue. Specifically, ...
Andrew Du's user avatar
3 votes

Do neural network weights need to add up to one?

do all the weights need to add up to 1.0 (i.e. 100%)? No. If not, why? Why should they? In the case of school grades, the justification might be that each component (midterm exam, homework etc.) is ...
Igor F.'s user avatar
  • 201
3 votes

Do neural network weights need to add up to one?

No, weights in that case do not constitute a formal probability distribution and do not need to be add up to 1. In fact, if they would — it could cause problems with underflow. Imagine a (nonsensical) ...
Scolpe's user avatar
  • 31
2 votes

Why are the initial weights of neural networks randomly initialised?

It is okay to initialize the weights to zero for a simple logistic regression, but for a neural network to initialize the weights to parameters to all zero and then apply gradient descent, it won't ...
ksgr5566's user avatar
  • 121
2 votes

How many parameter would there be in a logistic regression model used to classify reviews into "good" or "bad"?

Nope! Our number of coefficients will be driven by the vocabulary, and we'll use each of those 10K samples to estimate values for those coefficients - so, 'just' 100K samples. However, word ...
Edward Dixon's user avatar
2 votes

What do the neural network's weights represent conceptually?

I don't know if my intuition is correct but I will give it a try. You could see weights as how much important one thing is, the problem is to understand what that thing represents. When I say thing ...
gvgramazio's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible