# Tag Info

### In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the 3D filter (grey cube) is made up of a different value (...
• 982

### In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

In a convolutional neural network, is there a unique filter for each input channel or are the same new filters used across all input channels? The former. In fact there is a separate kernel defined ...
• 32.7k
Accepted

### Why are the initial weights of neural networks randomly initialised?

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue. http://www.deeplearningbook.org/contents/optimization.html Perhaps the only property known with complete ...
• 1,411
Accepted

### Do we know what the units of neural networks will do before we train them?

In reverse order to how you asked: all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal This actually happens if you ...
• 32.7k

### In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

I'm following up on the answers above with a concrete example in the hope to further clarify how the convolution works with respect to the input and output channels and the weights, respectively: Let ...
• 209

### Do neural network weights need to add up to one?

No, the weights do not need to add up to one. There isn't really a reason to do that. Weights as "contributions" may not be the best way to think about things here -- you're trying to learn ...
• 1,289

### Why are the initial weights of neural networks randomly initialised?

The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to ...

### Do neural network weights need to add up to one?

Nothing to do with it! Let me illustrate this concept using one of the simplest models in machine learning: Linear Regression. In linear regression, the goal is to find the coefficients of a ...
• 598
Accepted

### What is the significance of weights in a feedforward neural network?

You described a single-layer feedforward network. They can have multiple layers. The significance of the weights is that they make a linear transformation from the output of the previous layer and ...
• 391

### Why are the initial weights of neural networks randomly initialised?

That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, Gradient Descent Finds Global Minima of ...
• 725
Accepted

### Can neurons in MLP and filters in CNN be compared?

tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer Differences The neurons of these two types of layers have two key differences. These ...
• 3,193
Accepted

### What is the goal of weight initialization in neural networks?

Is it trying to make sure there is no symmetry in the gradients? The aim of weight initialization is to make sure that we don't converge to a trivial solution. That's why we have different kinds of ...
Accepted

### Is the bias also a "weight" in a neural network?

Yes, it is not unusual to omit the bias by adding a neuron which always outputs a constant 1, which will then be multiplied by an appropriate weight to give the same formula as you would get using an ...
• 1,010

### Can some of the weights be fixed during the training of a neural network?

Yes, you can fix (or freeze) some of the weights during the training of a neural network. In fact, this is done in the most common form of transfer learning (which is described here). I don't know ...
• 40.9k

### Is there a proper initialization technique for the weight matrices in multi-head attention?

IMO xavier/glorot is the correct way to initialize the $W_Q$ and $W_K$ matrices. In section 3.2.1 of the transformer paper the ...
• 815
Accepted

### How are non-linear surfaces formed in the training of a neural network?

Hi and welcome to the community. It's important to understand these basic concepts very clearly. You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let ...
• 148

### How are newer weight initialization techniques better than zero or random initialization?

There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, Xavier et al.. ...
• 381

### How does the neural-network know how to tweak weights for a specific neuron?

tl;dr The whole point of gradient descent is to assess the contribution of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter. A ...
• 3,193
Accepted

### Neural Nets: CNN confirming layer/filter arithmetic

Your first point is correct. The filters are stored in 4d arrays, with dimensions of (height, width, input channels, filter number) . The order may differ. Your second point is correct too. The ...
• 1,745

### In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

I recommend chapter 2.2.1 of my masters thesis as an answer. To add to the remaining answers: Keras is your friend to understand what happens: ...
• 1,055
Accepted

### Which hyperparameters in neural network are accesible to users adjustment

In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are: learning rate, ...
• 919
Accepted

### Not able to understand Pytorch Tensor (Weight & Biases) Size for Linear Regression

The size of the parameters tensor is depended on what type of layer that you want to build. Convolutional, fully connected, attention or even custom layer, each layer has a difference in the way it ...
• 912
Accepted

### Why and when do we need to normalize weights in Reinforcement Learning?

The kind of divergence that the other question experienced is a common problem with deep RL and temporal difference methods (Q-learning, SARSA, or any Actor Critic). The weight normalisation would not ...
• 32.7k

### What is actually being saved in the file when you save a model? For example a Tensorflow SavedModel file

Answer to Question 1 TensorFlow's documentation provides the following information on what is saved: The model config, weights, and optimizer are included in the SavedModel. Additionally, for every ...
• 1,939
Accepted

### Why does averaging attention-weighted positions reduce the effective resolution in transformers?

I think that the idea of "reduced effective resolution" from averaging is best understood through seeing how the proposed multi-head attention architecture fixes the issue. Specifically, ...

### Do neural network weights need to add up to one?

do all the weights need to add up to 1.0 (i.e. 100%)? No. If not, why? Why should they? In the case of school grades, the justification might be that each component (midterm exam, homework etc.) is ...
• 201

### Do neural network weights need to add up to one?

No, weights in that case do not constitute a formal probability distribution and do not need to be add up to 1. In fact, if they would — it could cause problems with underflow. Imagine a (nonsensical) ...
• 31

### Why are the initial weights of neural networks randomly initialised?

It is okay to initialize the weights to zero for a simple logistic regression, but for a neural network to initialize the weights to parameters to all zero and then apply gradient descent, it won't ...
• 121