Questions tagged [weights]

For questions about the concept of weight (or parameter) of a machine learning model, such as a neural network or a linear regression model.

Filter by
Sorted by
Tagged with
0 votes
0 answers
22 views

What do "large variables" and "small weights" mean in these sentences?

I'm trying to understand these two points from an article: Models with large variables i.e weight matrices. As a consequence such models have correspondingly large gradients and optimizer states. The ...
user avatar
  • 101
0 votes
1 answer
93 views

Why do neural network weights have to be between 0 and 1?

I've been reading about neural networks for a long time, and I saw that in each one, the weights are always between 0 and 1. Why is this? I tried programming one, but the sigmoid function just seemed ...
user avatar
0 votes
0 answers
23 views

Why do I have a shape mismatch when running back propagation of convolutional layer in CNN?

I am having problems with my array shapes when calculating change in loss wrt Kernel (delL/delk) in back propagation of convolutional layer. I am running a mini batch neural network operating on 32 ...
user avatar
  • 11
1 vote
1 answer
56 views

Why doesn't the high precision of neural network weights improve accuracy?

Consider the following paragraph from the subsubsection 3.5.2: A dtype for every occasion chapter named It starts with a tensor from the textbook titled Deep Learning with PyTorch by Eli Stevens et al....
user avatar
  • 2,975
0 votes
1 answer
49 views

Is there any advantage in viewing weights of a neural network as random variables?

In artificial intelligence, especially in machine learning, the inputs and outputs of neurons in a neural network can be viewed as random variables. And this view is highly useful in many ways. The ...
user avatar
  • 2,975
0 votes
1 answer
56 views

How many layers and neurons in a FFNN do I need to make it equivalent to a CNN?

I started to learn machine learning early, and I studied the convolutional neural network and its ability to understand images and how it helps to reduce the number of parameters that need to be tuned....
user avatar
0 votes
1 answer
115 views

Not able to understand Pytorch Tensor (Weight & Biases) Size for Linear Regression

Below are the two tensors ...
user avatar
  • 111
0 votes
0 answers
14 views

What to infer If a parameter value is the same as I initialized while training?

I am running a neural network model. There are several parameters. I initialized a new parameter to a value (say k). When I observe its change during the training, it is not changing like other ...
user avatar
  • 2,975
1 vote
0 answers
10 views

How does the distribution of the parameters change in logistic regression?

I have my own data to train a logistic regression model (for a multi-class classification task), and I want to know how the distribution of weight parameters changes after each update with gradient ...
user avatar
1 vote
0 answers
12 views

Are there any recommendations on initialising a single parameter in deep learning?

I want to initialize a parameter, which is a single real number in my model. If you want the role of the parameter in the model, you can assume it as the parameter to multiply with the output of the ...
user avatar
  • 2,975
3 votes
0 answers
64 views

Are there neural networks with (hard) constraints on the weights?

I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For ...
user avatar
  • 163
1 vote
1 answer
32 views

Rank of gradient-of-loss with respect to layer weights in an MLP

The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3: The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch ...
user avatar
  • 63
4 votes
1 answer
56 views

In TD(0) with linear function approximation, why is the gradient of $\hat v(S^{\prime}, \mathbf w)$ wrt parameters $\mathbf w$ not considered?

I am reading these slides. On page 38, the update for the parameters for the linear function approximation of TD(0) is given. I have a doubt regarding this. The cost function (RMSE) is given on page ...
user avatar
-2 votes
1 answer
34 views

model and trained model parameters on CIFAR-10 [closed]

I'm looking for different models (specifically ResNet18/20, ResNet32/34, VGG16, MobileNet and SqueezeNet) and their parameters after training (i.e., .pth file) that ...
user avatar
1 vote
1 answer
53 views

Is there a way to update the neural network to fit the new data without the time required for retraining?

I built a basic neural network in MATLAB. The neural network classifies points on the X-Y axis system into two classes (0 and 1). (I try to get the function that represents a shape from this photo) ...
user avatar
0 votes
0 answers
62 views

What would be the total number of learnable parameters of the RNN encoder of this encoder-decoder architecture for machine translation?

Here's a quiz. My answer is different from the teacher's, so I'm wondering what answer would you pick up. We use a sequence-to-sequence (encoder-decoder) system to perform machine translation. We ...
user avatar
  • 113
0 votes
0 answers
41 views

In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?

Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
user avatar
0 votes
2 answers
56 views

Correctly input additional values into CNN

I understand that in order to add additional inputs to a CNN, e.g. in self driving, I can append the data to a flattened layer after the convolutions and before the fully connected layers. However, a ...
user avatar
  • 101
0 votes
0 answers
41 views

Using parameter estimation for training a neural network

Assume that we have 4 layers in a neural network. $$z_1 = L_1(x, W_1)$$ $$z_2 = L_2(z_1, W_2)$$ $$z_3 = L_3(z_2, W_3)$$ $$y = L_1(z_3, W_4)$$ Where $x$ is the vector input, $y$ is the vector output ...
user avatar
  • 143
0 votes
0 answers
28 views

How did authors ensure that critical points do exist in GAN?

Using an MLP as a generator introduces multiple critical points in parameter space. You can read this excerpt from the research paper titled Generative Adversarial Nets by Ian J. Goodfellow et al. In ...
user avatar
  • 2,975
1 vote
0 answers
75 views

Is there a proper initialization technique for the weight matrices in multi-head attention?

Self-attention layers have 4 learnable tensors (in the vanilla formulation): Query matrix $W_Q$ Key matrix $W_K$ Value matrix $W_V$ Output matrix $W_O$ Nice illustration from https://jalammar....
user avatar
0 votes
0 answers
7 views

Are there any connectionist parametric models with non-neuron building blocks?

Parametric models allows learning by converging to the desired parameters, which are randomly initialized initially. Among the parametric models, especially in connectionist AI, neural networks are ...
user avatar
  • 2,975
0 votes
0 answers
24 views

How to handle critical points during generator training?

Using an MLP as a generator introduces multiple critical points in parameter space. You can read this excerpt from the research paper titled Generative Adversarial Nets by Ian J. Goodfellow et al. In ...
user avatar
  • 2,975
1 vote
0 answers
20 views

Recursive Least squares (RLS) for mini batch

For my application I am considering a learning problem where I simulate a bunch of episodes say '$n$' first, and than carry out the recursive least squares update. Similar to $TD(1)$. I know that RLS ...
user avatar
0 votes
1 answer
64 views

What are the applications in which the precision of the neural network's weights is unimportant?

While reading about Module in PyTorch, I came across a new data type called half datatype. half() method when calls on a Module ...
user avatar
  • 2,975
0 votes
0 answers
56 views

What are the numbers that are useful (may need to be stored) other than parameters of a model?

Consider the following method related to buffers in PyTorch ...
user avatar
  • 2,975
3 votes
1 answer
121 views

Can some of the weights be fixed during the training of a neural network?

Is it possible to exclude specific layers from the optimization? For example, let's say I have an input layer, 2 hidden layers, and the output layer. I know there is a perfect solution for my problem ...
user avatar
0 votes
0 answers
16 views

Prioritized Experience Replay, clarifications for Important Sampling

I can't seem to understand how the weight equation is dissected and how it really works when combined with the TD-error value. The weight equation is: I can understand what N, P(i) and beta represent,...
user avatar
0 votes
1 answer
63 views

Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
user avatar
0 votes
0 answers
338 views

If I have a feature matrix of shape $1200 \times 2000$, should I have $2000$ input neurons?

I have an input feature matrix (row-normalized) of size $1200 \times 2000$. When I am writing my neural network, is the number of input neurons equal to the number of columns, that is $2000$? Is it ...
user avatar
1 vote
1 answer
33 views

Why are weights not initialized with mean=1?

I wonder why weights are initialized with zero-mean. It is one of the reasons, why deep architectures cannot be trained without skip connections. Without the skip connections, the zero initialization ...
user avatar
  • 31
1 vote
0 answers
22 views

Which methods for weight initialization in Neural Networks are currently common practice?

I am currently researching the topic of weight initialization methods for (deep) neural networks and I'm a little stuck. The result of my work should be an overview of methods that are currently in ...
user avatar
  • 11
3 votes
1 answer
242 views

Is the bias also a "weight" in a neural network?

I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a "weight" and a "...
user avatar
2 votes
1 answer
63 views

Is this LSTM layer learning anything?

I've trained a CNN-LSTM model but the results weren't satisfactory, so I took a look at my weight distributions and this is what I got: I don't understand. Is this layer learning anything? Or no? ...
user avatar
2 votes
1 answer
54 views

How to train a neural network with few weights and biases held constant?

I am a beginner in neural networks. I am building a neural network with 3 layers. The input $X$ has 7 features and the output $Y$ is a real number. In the hidden layer, there are two nodes. The bottom ...
user avatar
  • 21
5 votes
1 answer
162 views

Why did the developement of neural networks stop between 50s and 80s?

In a video lecture on the development of neural networks and the history of deep learning (you can start from minute 13), the lecturer (Yann LeCunn) said that the development of neural networks ...
user avatar
  • 489
2 votes
1 answer
53 views

Does the weight vector form imply feature space curvature?

I came across this sentence when exploring a simple nearest neighbor classifier method using Euclidean distance (link): The slightly odd thing about using the Euclidean distance to compare features ...
user avatar
1 vote
1 answer
116 views

Is there a convention on the order of multiplication of the weights with the inputs in neural nets?

Is there a convention on how the input data and the weights are multiplied? The input data can be anything, including the result from the previous layers. There are two options: Option 1: $$\begin{...
user avatar
  • 65
4 votes
0 answers
69 views

Why does sigmoid saturation prevent signal flow through the neuron?

As per these slides on page 35: Sigmoids saturate and kill gradients. when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. the gradient and ...
user avatar
  • 183
2 votes
0 answers
82 views

Are there any new weight initialization techniques for DNN published after 2015?

Considering weights initialization in my personal projects, I always used some standard techniques such as: Glorot (also known as Xavier) initialization (2010). Mertens initialization (2010). He ...
user avatar
1 vote
1 answer
58 views

What are acting as weights in a convolution neural network?

Looking at some old notes I took on CNN's and I wrote down that the weights in a CNN are acting like filters in a CNN but to be honest I don't really know what the weights are acting as in a CNN and ...
user avatar
0 votes
0 answers
32 views

How do we interpret the images of weights in logistic regression

The following images are a) The weights of a logistic regression model trained on MNIST. b) The sign of the weights of a logistic regression How do these images represent the weights? Would be ...
user avatar
  • 159
0 votes
1 answer
43 views

What does "adding class weights for an imbalanced dataset" mean in the case of multi-label classification?

Suppose I have the following toy data set: Each instance has multiple labels at a time. You can see I have 2 instances for Label2. However, only one instance for the other labels. It means that we ...
user avatar
4 votes
0 answers
52 views

When is using weight regularization bad?

Regularization of weights (e.g. L1 or L2) keeps them small and standardized, which can help reduce data overfitting. From this article, regularization sounds favorable in many cases, but is it always ...
user avatar
  • 663
1 vote
0 answers
63 views

Is the performance of a neural network, which was trained with encrypted data and weights, affected if the weights are decrypted?

Suppose that a neural network is trained with encrypted (for example, with homomorphic encryption and, more precisely, with the Paillier partial scheme) data. Moreover, suppose that it is also trained ...
user avatar
  • 73
5 votes
0 answers
79 views

Why is there a Uniform and Normal version of He / Xavier initialization in DL libraries?

Two of the most popular initialization schemes for neural network weights today are Xavier and He. Both methods propose random weight initialization with a variance dependent on the number of input ...
user avatar
  • 600
1 vote
1 answer
219 views

In attention models with multiple layers, are weight matrices shared across layers?

In articles that describe neural architectures with multiple attention layers of the same form, are the weight matrices usually the same across the layers? Consider as an example, "Attention is ...
user avatar
  • 138
1 vote
2 answers
171 views

In the multi-head attention mechanism of the transformer, why do we need both $W_i^Q$ and ${W_i^K}^T$?

In the Attention is all you need paper, on the 4th page, we have equation 1, which describes the self-attention mechanism of the transformer architecture $$ \text { Attention }(Q, K, V)=\operatorname{...
user avatar
1 vote
0 answers
91 views

Why should variance(output) equal variance(input) in Xavier Initialisation?

In a lot of explanations online for Xavier Initialization, I see the following: With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
user avatar
3 votes
0 answers
51 views

How efficient is SCAWI weight initialization method?

I'm currently in the middle of a project (for my thesis) constructing a deep neural network. Since I'm still in the research part, I'm trying to find various ways and techniques to initialize weights. ...
user avatar
  • 41