Questions tagged [weights]
For questions about the concept of weight (or parameter) of a machine learning model, such as a neural network or a linear regression model.
86
questions
0
votes
0
answers
29
views
Can you illustrate how the weights in transformer model generated from a training sentence can be generalized to an unseen test sentence?
Can you show how the weights in transformer model are generalizable?
- 99
7
votes
4
answers
4k
views
Do neural network weights need to add up to one?
The idea that weights determine how much influence each input value from the current layer will have when calculating the input to the following layer reminds me of when my professors would say that ...
- 181
1
vote
0
answers
36
views
Why does averaging attention-weighted positions reduce the effective resolution in transformers?
I was reading this blog post from Harvard and it says in its background paragraph about transformers that the number of operations required to relate signals from two arbitrary input or output ...
- 573
0
votes
1
answer
68
views
What is actually being saved in the file when you save a model? For example a Tensorflow SavedModel file [closed]
I'm building a feature for my application that requires reading the properties of a saved ML model file (after it's trained). However, as I am pretty new to this field, I don't really understand the ...
- 113
0
votes
1
answer
86
views
Does it make sense to store information in a variable defined inside a Pytorch nn.Module?
I have a pytorch model (custom model inherited from nn.Module).
I'm developing some architecture, for which makes sense for my task to have a list defined in the model as:
...
0
votes
1
answer
313
views
How is InstructGPT a fine-tuned version of GPT-3 and at the same time has fewer parameters than the original GPT3?
I am reading the paper "Training language models to follow instructions with human feedback"
It says:
Our labelers provide demonstrations of the desired behavior on the input prompt ...
0
votes
0
answers
19
views
Should we forget last weight update in neural network?
When training a neural network, the general process could be something like this:
While error < min_error
Forward pass
Compute error and cost funcion
Back propagation
Update weights
But when we ...
- 1
1
vote
0
answers
17
views
Giving Specified Data a Larger Value/Weight in a Model
I'm in the process of creating a model to classify an occupational code based on a job title & description. I have a large sample of labelled data to achieve this.
The government has a resource ...
- 11
0
votes
1
answer
74
views
Why the x-input should be multiplied by weight in an artificial neuron?
So why weight should be multiplied with input?
Yes I know, weight is intended for tuning the connection strength of input that will affect output so that's will be useful for learning (CMIIW).
But why ...
0
votes
1
answer
529
views
Is there a way to freeze training for weights, but not biases in PyTorch? [closed]
I'm constructing a neural network where the weights of my first hidden layer (connected to the input) are all 1 (identity matrix), but the biases are variable.
Is there a way to "freeze" any ...
- 133
1
vote
1
answer
113
views
How to calculate the total number of inputs in CNN?
I search this kind of question for a while and I find many discussions involve on counting the number of parameters of a Convolutional Neural Network, but not on the inputs. Using the Fashion MNIST ...
- 11
1
vote
0
answers
41
views
Do NNs suffer from lack of efficiency in network structure and suggesting training parameters?
I am working on dynamical systems using Optimal Control theory and trying to find the connection between this field and Machine Learning. Consider a simple 2-layer Neural Network (NN) where the ...
0
votes
1
answer
133
views
Why and when do we need to normalize weights in Reinforcement Learning?
I recently came across this SO question, wherein the poster was asked to normalize their weights while using a function approximator with SARSA. I don't remember having to normalize any weights while ...
- 618
1
vote
0
answers
36
views
What do "large variables" and "small weights" mean in these sentences?
I'm trying to understand these two points from an article:
Models with large variables i.e weight matrices. As a consequence such models have correspondingly large gradients and optimizer states. The ...
- 111
1
vote
1
answer
1k
views
Why do neural network weights have to be between 0 and 1?
I've been reading about neural networks for a long time, and I saw that in each one, the weights are always between 0 and 1. Why is this? I tried programming one, but the sigmoid function just seemed ...
- 137
2
votes
1
answer
475
views
Why doesn't the high precision of neural network weights improve accuracy?
Consider the following paragraph from the subsubsection 3.5.2: A dtype for every occasion chapter named It starts with a tensor from the textbook titled Deep Learning with PyTorch by Eli Stevens et al....
- 3,612
0
votes
1
answer
362
views
Is there any advantage in viewing weights of a neural network as random variables?
In artificial intelligence, especially in machine learning, the inputs and outputs of neurons in a neural network can be viewed as random variables.
And this view is highly useful in many ways. The ...
- 3,612
0
votes
1
answer
120
views
How many layers and neurons in a FFNN do I need to make it equivalent to a CNN?
I started to learn machine learning early, and I studied the convolutional neural network and its ability to understand images and how it helps to reduce the number of parameters that need to be tuned....
1
vote
1
answer
699
views
Not able to understand Pytorch Tensor (Weight & Biases) Size for Linear Regression
Below are the two tensors
...
- 121
1
vote
0
answers
13
views
How does the distribution of the parameters change in logistic regression?
I have my own data to train a logistic regression model (for a multi-class classification task), and I want to know how the distribution of weight parameters changes after each update with gradient ...
- 173
1
vote
0
answers
18
views
Are there any recommendations on initialising a single parameter in deep learning?
I want to initialize a parameter, which is a single real number in my model. If you want the role of the parameter in the model, you can assume it as the parameter to multiply with the output of the ...
- 3,612
3
votes
0
answers
253
views
Are there neural networks with (hard) constraints on the weights?
I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For ...
- 173
1
vote
1
answer
70
views
Rank of gradient-of-loss with respect to layer weights in an MLP
The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3:
The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch ...
- 63
5
votes
1
answer
149
views
In TD(0) with linear function approximation, why is the gradient of $\hat v(S^{\prime}, \mathbf w)$ wrt parameters $\mathbf w$ not considered?
I am reading these slides. On page 38, the update for the parameters for the linear function approximation of TD(0) is given. I have a doubt regarding this.
The cost function (RMSE) is given on page ...
- 53
-2
votes
1
answer
155
views
model and trained model parameters on CIFAR-10 [closed]
I'm looking for different models (specifically ResNet18/20, ResNet32/34, VGG16, MobileNet and SqueezeNet) and their parameters after training (i.e., .pth file) that ...
3
votes
1
answer
145
views
Is there a way to update the neural network to fit the new data without the time required for retraining?
I built a basic neural network in MATLAB. The neural network classifies points on the X-Y axis system into two classes (0 and 1).
(I try to get the function that represents a shape from this photo)
...
- 39
0
votes
0
answers
470
views
In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?
Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
0
votes
2
answers
84
views
Correctly input additional values into CNN
I understand that in order to add additional inputs to a CNN, e.g. in self driving, I can append the data to a flattened layer after the convolutions and before the fully connected layers.
However, a ...
- 101
0
votes
0
answers
122
views
Using parameter estimation for training a neural network
Assume that we have 4 layers in a neural network.
$$z_1 = L_1(x, W_1)$$
$$z_2 = L_2(z_1, W_2)$$
$$z_3 = L_3(z_2, W_3)$$
$$y = L_1(z_3, W_4)$$
Where $x$ is the vector input, $y$ is the vector output ...
- 143
7
votes
1
answer
3k
views
Is there a proper initialization technique for the weight matrices in multi-head attention?
Self-attention layers have 4 learnable tensors (in the vanilla formulation):
Query matrix $W_Q$
Key matrix $W_K$
Value matrix $W_V$
Output matrix $W_O$
Nice illustration from https://jalammar....
- 2,564
1
vote
0
answers
36
views
Recursive Least squares (RLS) for mini batch
For my application I am considering a learning problem where I simulate a bunch of episodes say '$n$' first, and than carry out the recursive least squares update. Similar to $TD(1)$.
I know that RLS ...
0
votes
2
answers
92
views
What are the applications in which the precision of the neural network's weights is unimportant?
While reading about Module in PyTorch, I came across a new data type called half datatype.
half() method when calls on a Module ...
- 3,612
0
votes
0
answers
61
views
What are the numbers that are useful (may need to be stored) other than parameters of a model?
Consider the following method related to buffers in PyTorch
...
- 3,612
3
votes
1
answer
852
views
Can some of the weights be fixed during the training of a neural network?
Is it possible to exclude specific layers from the optimization?
For example, let's say I have an input layer, 2 hidden layers, and the output layer. I know there is a perfect solution for my problem ...
- 43
0
votes
1
answer
162
views
Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?
I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
- 1
1
vote
1
answer
49
views
Why are weights not initialized with mean=1?
I wonder why weights are initialized with zero-mean. It is one of the reasons, why deep architectures cannot be trained without skip connections. Without the skip connections, the zero initialization ...
- 31
1
vote
0
answers
30
views
Which methods for weight initialization in Neural Networks are currently common practice?
I am currently researching the topic of weight initialization methods for (deep) neural networks and I'm a little stuck. The result of my work should be an overview of methods that are currently in ...
- 11
3
votes
1
answer
986
views
Is the bias also a "weight" in a neural network?
I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a "weight" and a "...
- 206
2
votes
1
answer
117
views
Is this LSTM layer learning anything?
I've trained a CNN-LSTM model but the results weren't satisfactory, so I took a look at my weight distributions and this is what I got:
I don't understand. Is this layer learning anything? Or no?
...
2
votes
1
answer
123
views
How to train a neural network with few weights and biases held constant?
I am a beginner in neural networks. I am building a neural network with 3 layers. The input $X$ has 7 features and the output $Y$ is a real number. In the hidden layer, there are two nodes. The bottom ...
- 21
5
votes
1
answer
321
views
Why did the developement of neural networks stop between 50s and 80s?
In a video lecture on the development of neural networks and the history of deep learning (you can start from minute 13), the lecturer (Yann LeCunn) said that the development of neural networks ...
- 573
2
votes
1
answer
63
views
Does the weight vector form imply feature space curvature?
I came across this sentence when exploring a simple nearest neighbor classifier method using Euclidean distance (link):
The slightly odd thing about using the Euclidean distance to compare features ...
- 21
1
vote
1
answer
382
views
Is there a convention on the order of multiplication of the weights with the inputs in neural nets?
Is there a convention on how the input data and the weights are multiplied? The input data can be anything, including the result from the previous layers.
There are two options:
Option 1:
$$\begin{...
- 85
4
votes
0
answers
151
views
Why does sigmoid saturation prevent signal flow through the neuron?
As per these slides on page 35:
Sigmoids saturate and kill gradients.
when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero.
the gradient and ...
- 193
2
votes
0
answers
134
views
Are there any new weight initialization techniques for DNN published after 2015?
Considering weights initialization in my personal projects, I always used some standard techniques such as:
Glorot (also known as Xavier) initialization (2010).
Mertens initialization (2010).
He ...
- 183
1
vote
1
answer
1k
views
What are acting as weights in a convolution neural network?
Looking at some old notes I took on CNN's and I wrote down that the weights in a CNN are acting like filters in a CNN but to be honest I don't really know what the weights are acting as in a CNN and ...
0
votes
0
answers
76
views
How do we interpret the images of weights in logistic regression
The following images are
a) The weights of a logistic regression model trained on MNIST.
b) The sign of the weights of a logistic regression
How do these images represent the weights?
Would be ...
- 159
0
votes
1
answer
271
views
What does "adding class weights for an imbalanced dataset" mean in the case of multi-label classification?
Suppose I have the following toy data set:
Each instance has multiple labels at a time.
You can see I have 2 instances for Label2. However, only one instance for the other labels. It means that we ...
- 41
4
votes
0
answers
62
views
When is using weight regularization bad?
Regularization of weights (e.g. L1 or L2) keeps them small and standardized, which can help reduce data overfitting. From this article, regularization sounds favorable in many cases, but is it always ...
- 763
1
vote
0
answers
65
views
Is the performance of a neural network, which was trained with encrypted data and weights, affected if the weights are decrypted?
Suppose that a neural network is trained with encrypted (for example, with homomorphic encryption and, more precisely, with the Paillier partial scheme) data. Moreover, suppose that it is also trained ...
- 73