27
votes
In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?
The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the 3D filter (grey cube) is made up of a different value (...
- 952
15
votes
In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?
In a convolutional neural network, is there a unique filter for each input channel or are the same new filters used across all input channels?
The former. In fact there is a separate kernel defined ...
- 26.2k
13
votes
Accepted
Why are the initial weights of neural networks randomly initialised?
You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue.
http://www.deeplearningbook.org/contents/optimization.html
Perhaps the only property known with complete ...
- 1,411
8
votes
Accepted
Do we know what the units of neural networks will do before we train them?
In reverse order to how you asked:
all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal
This actually happens if you ...
- 26.2k
7
votes
In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?
I'm following up on the answers above with a concrete example in the hope to further clarify how the convolution works with respect to the input and output channels and the weights, respectively:
Let ...
- 199
7
votes
Why are the initial weights of neural networks randomly initialised?
The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to ...
- 853
5
votes
Accepted
What is the significance of weights in a feedforward neural network?
You described a single-layer feedforward network. They can have multiple layers. The significance of the weights is that they make a linear transformation from the output of the previous layer and ...
- 381
4
votes
Accepted
Can neurons in MLP and filters in CNN be compared?
tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer
Differences
The neurons of these two types of layers have two key differences. These ...
- 3,143
4
votes
Can some of the weights be fixed during the training of a neural network?
Yes, you can fix (or freeze) some of the weights during the training of a neural network. In fact, this is done in the most common form of transfer learning (which is described here). I don't know ...
- 37k
3
votes
Accepted
How are non-linear surfaces formed in the training of a neural network?
Hi and welcome to the community. It's important to understand these basic concepts very clearly.
You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let ...
- 148
3
votes
How are newer weight initialization techniques better than zero or random initialization?
There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, Xavier et al.. ...
- 381
3
votes
How does the neural-network know how to tweak weights for a specific neuron?
tl;dr
The whole point of gradient descent is to assess the contribution of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter.
A ...
- 3,143
3
votes
Accepted
Neural Nets: CNN confirming layer/filter arithmetic
Your first point is correct. The filters are stored in 4d arrays, with dimensions of (height, width, input channels, filter number) . The order may differ.
Your second point is correct too. The ...
- 1,725
3
votes
Why are the initial weights of neural networks randomly initialised?
That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, Gradient Descent Finds Global Minima of ...
- 675
3
votes
Accepted
What is the goal of weight initialization in neural networks?
Is it trying to make sure there is no symmetry in the gradients?
The aim of weight initialization is to make sure that we don't converge to a trivial solution. That's why we have different kinds of ...
- 728
3
votes
Accepted
Which hyperparameters in neural network are accesible to users adjustment
In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are:
learning rate,
...
- 909
3
votes
Accepted
Is the bias also a "weight" in a neural network?
Yes, it is not unusual to omit the bias by adding a neuron which always outputs a constant 1, which will then be multiplied by an appropriate weight to give the same formula as you would get using an ...
- 1,000
3
votes
Accepted
Why and when do we need to normalize weights in Reinforcement Learning?
The kind of divergence that the other question experienced is a common problem with deep RL and temporal difference methods (Q-learning, SARSA, or any Actor Critic).
The weight normalisation would not ...
- 26.2k
2
votes
Why are the initial weights of neural networks randomly initialised?
It is okay to initialize the weights to zero for a simple logistic regression, but for a neural network to initialize the weights to parameters to all zero and then apply gradient descent, it won't ...
- 21
2
votes
In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?
I recommend chapter 2.2.1 of my masters thesis as an answer. To add to the remaining answers:
Keras is your friend to understand what happens:
...
- 1,025
2
votes
How many parameter would there be in a logistic regression model used to classify reviews into "good" or "bad"?
Nope! Our number of coefficients will be driven by the vocabulary, and we'll use each of those 10K samples to estimate values for those coefficients - so, 'just' 100K samples. However, word ...
- 327
2
votes
What do the neural network's weights represent conceptually?
I don't know if my intuition is correct but I will give it a try.
You could see weights as how much important one thing is, the problem is to understand what that thing represents. When I say thing ...
- 666
2
votes
Accepted
Classification with deeplearning : clean start vs continue training
If the task involves only apples, orange and peaches, you should use method 1. As the number of classes is small, the network cannot generalize well to all classes. As a side note, you should start ...
- 1,725
2
votes
Accepted
When would bias regularisation and activation regularisation be necessary?
Regularizer's are used as a means to combat over fitting.They essentially create a cost function penalty which tries to prevent quantities from becoming to large. I have primarily used kernel ...
- 694
2
votes
Accepted
What are some examples of functions that machine learning models compute?
If I understood correctly, the model is a polynomial equation
No, it's not true that all machine learning (ML) models compute (or represent) a polynomial function. For example, a sigmoid is not a ...
- 37k
2
votes
Accepted
Should the range and initial values of weights and biases be adjusted to fit input and output data?
is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well?
No; and it is no coincidence that you will not be able to find any reference to ...
- 979
2
votes
Is there a convention on the order of multiplication of the weights with the inputs in neural nets?
The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention.
Some examples include:
In Deep Learning: ...
- 1,000
2
votes
Why are weights not initialized with mean=1?
Interesting question,
I can come with 2 explanations why we don't initialize weights with 1 mean value :
It may be easier for the network to learn identity function, but we may have a similar issue ...
- 211
2
votes
Accepted
Not able to understand Pytorch Tensor (Weight & Biases) Size for Linear Regression
The size of the parameters tensor is depended on what type of layer that you want to build. Convolutional, fully connected, attention or even custom layer, each layer has a difference in the way it ...
- 857
2
votes
Is there any advantage in viewing weights of a neural network as random variables?
In Bayesian statistics, as opposed to frequentist statistics, you can model the parameters as random variables. Bayesian machine learning is the application of Bayesian statistics in the context of ...
- 37k
Only top scored, non community-wiki answers of a minimum length are eligible