Hot answers tagged

25 votes

In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the 3D filter (grey cube) is made up of a different value (...
user avatar
15 votes

In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

In a convolutional neural network, is there a unique filter for each input channel or are the same new filters used across all input channels? The former. In fact there is a separate kernel defined ...
user avatar
  • 23.9k
13 votes
Accepted

Why are the initial weights of neural networks randomly initialised?

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue. http://www.deeplearningbook.org/contents/optimization.html Perhaps the only property known with complete ...
user avatar
  • 1,390
8 votes
Accepted

Do we know what the units of neural networks will do before we train them?

In reverse order to how you asked: all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal This actually happens if you ...
user avatar
  • 23.9k
7 votes

In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

I'm following up on the answers above with a concrete example in the hope to further clarify how the convolution works with respect to the input and output channels and the weights, respectively: Let ...
user avatar
  • 199
7 votes

Why are the initial weights of neural networks randomly initialised?

The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to ...
user avatar
5 votes
Accepted

What is the significance of weights in a feedforward neural network?

You described a single-layer feedforward network. They can have multiple layers. The significance of the weights is that they make a linear transformation from the output of the previous layer and ...
user avatar
  • 381
4 votes

Can some of the weights be fixed during the training of a neural network?

Yes, you can fix (or freeze) some of the weights during the training of a neural network. In fact, this is done in the most common form of transfer learning (which is described here). I don't know ...
user avatar
  • 33.8k
3 votes
Accepted

Which hyperparameters in neural network are accesible to users adjustment

In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are: learning rate, ...
user avatar
  • 889
3 votes
Accepted

What is the goal of weight initialization in neural networks?

Is it trying to make sure there is no symmetry in the gradients? The aim of weight initialization is to make sure that we don't converge to a trivial solution. That's why we have different kinds of ...
user avatar
3 votes
Accepted

Can neurons in MLP and filters in CNN be compared?

tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer Differences The neurons of these two types of layers have two key differences. These ...
user avatar
  • 3,093
3 votes

How are newer weight initialization techniques better than zero or random initialization?

There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, Xavier et al.. ...
user avatar
  • 381
3 votes
Accepted

How are non-linear surfaces formed in the training of a neural network?

Hi and welcome to the community. It's important to understand these basic concepts very clearly. You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let ...
user avatar
  • 128
3 votes
Accepted

Neural Nets: CNN confirming layer/filter arithmetic

Your first point is correct. The filters are stored in 4d arrays, with dimensions of (height, width, input channels, filter number) . The order may differ. Your second point is correct too. The ...
user avatar
  • 1,715
3 votes

How does the neural-network know how to tweak weights for a specific neuron?

tl;dr The whole point of gradient descent is to assess the contribution of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter. A ...
user avatar
  • 3,093
3 votes
Accepted

Is the bias also a "weight" in a neural network?

Yes, it is not unusual to omit the bias by adding a neuron which always outputs a constant 1, which will then be multiplied by an appropriate weight to give the same formula as you would get using an ...
user avatar
  • 941
3 votes
Accepted

Why and when do we need to normalize weights in Reinforcement Learning?

The kind of divergence that the other question experienced is a common problem with deep RL and temporal difference methods (Q-learning, SARSA, or any Actor Critic). The weight normalisation would not ...
user avatar
  • 23.9k
2 votes

What do the neural network's weights represent conceptually?

I don't know if my intuition is correct but I will give it a try. You could see weights as how much important one thing is, the problem is to understand what that thing represents. When I say thing ...
user avatar
2 votes

Why are the initial weights of neural networks randomly initialised?

That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, Gradient Descent Finds Global Minima of ...
user avatar
2 votes
Accepted

Classification with deeplearning : clean start vs continue training

If the task involves only apples, orange and peaches, you should use method 1. As the number of classes is small, the network cannot generalize well to all classes. As a side note, you should start ...
user avatar
  • 1,715
2 votes

How many parameter would there be in a logistic regression model used to classify reviews into "good" or "bad"?

Nope! Our number of coefficients will be driven by the vocabulary, and we'll use each of those 10K samples to estimate values for those coefficients - so, 'just' 100K samples. However, word ...
user avatar
2 votes
Accepted

Should the range and initial values of weights and biases be adjusted to fit input and output data?

is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well? No; and it is no coincidence that you will not be able to find any reference to ...
user avatar
2 votes
Accepted

When would bias regularisation and activation regularisation be necessary?

Regularizer's are used as a means to combat over fitting.They essentially create a cost function penalty which tries to prevent quantities from becoming to large. I have primarily used kernel ...
user avatar
  • 634
2 votes
Accepted

What are some examples of functions that machine learning models compute?

If I understood correctly, the model is a polynomial equation No, it's not true that all machine learning (ML) models compute (or represent) a polynomial function. For example, a sigmoid is not a ...
user avatar
  • 33.8k
2 votes

Is there a convention on the order of multiplication of the weights with the inputs in neural nets?

The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention. Some examples include: In Deep Learning: ...
user avatar
  • 941
2 votes

Why are weights not initialized with mean=1?

Interesting question, I can come with 2 explanations why we don't initialize weights with 1 mean value : It may be easier for the network to learn identity function, but we may have a similar issue ...
user avatar
  • 211
2 votes
Accepted

Not able to understand Pytorch Tensor (Weight & Biases) Size for Linear Regression

The size of the parameters tensor is depended on what type of layer that you want to build. Convolutional, fully connected, attention or even custom layer, each layer has a difference in the way it ...
user avatar
  • 817
2 votes

Is there any advantage in viewing weights of a neural network as random variables?

In Bayesian statistics, as opposed to frequentist statistics, you can model the parameters as random variables. Bayesian machine learning is the application of Bayesian statistics in the context of ...
user avatar
  • 33.8k
2 votes

Why doesn't the high precision of neural network weights improve accuracy?

First, I have not read and do not have that book. That said, I would interpret that statement in the context of the intractability of guaranteeing that the optimization function will find global ...
user avatar
1 vote

What does "adding class weights for an imbalanced dataset" mean in the case of multi-label classification?

The paper A systematic study of the class imbalance problem in convolutional neural networks is a great overview on class imbalance approaches. Section 2 summarizes various methods commonly used. They ...
user avatar
  • 1,230

Only top scored, non community-wiki answers of a minimum length are eligible