Share Your Experience: Take the 2024 Developer Survey

# Questions tagged [weights]

For questions about the concept of weight (or parameter) of a machine learning model, such as a neural network or a linear regression model.

91 questions
Filter by
Sorted by
Tagged with
5 views

### Apply weight to one feature based on another one for training a regression model

I have 1000 items that have a numerical feature y, the ground truth that I want to predict. Each item has another feature c that ...
22 views

### Why are the Q and K matrices two separate matrices in attention?

If I understand correctly the attention layer is represented as  \begin{align} &softmax(\frac{Q K^T}{\sqrt{d_k}}) V \\ = &softmax(\frac{(s W_q) (s W_k)^T}{\sqrt{d_k}}) V \\ = &softmax(\...
1 vote
84 views

### Rank of gradient-of-loss with respect to layer weights in an MLP

The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3: The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch ...
5 views

### Is it possible to apply transfer learning between Temporal Fusion Transformer and sequential architecture LSTM and GRU

If TFT is a pretrained model, is it possible to transfer the weights to sequential neural network models like LSTM,BILSTM and GRU.
1k views

### How are weights for weighted x-entropy loss on imbalanced data calculated?

I am trying to build a classifier which should be trained with the cross entropy loss. The training data is highly class-imbalanced. To tackle this, I've gone through the advice of the tensorflow docs ...
163 views

### Is there a way to update the neural network to fit the new data without the time required for retraining?

I built a basic neural network in MATLAB. The neural network classifies points on the X-Y axis system into two classes (0 and 1). (I try to get the function that represents a shape from this photo) ...
15 views

### Might use of rational numbers and calculations be beneficial for an ANN?

Rational numbers would help alleviate some gradient issues by not losing precision as the weights and the propagated values (signal) reach extremely low and high values. I'm not aware of any hardware ...
1 vote
97 views

### Why does averaging attention-weighted positions reduce the effective resolution in transformers?

I was reading this blog post from Harvard and it says in its background paragraph about transformers that the number of operations required to relate signals from two arbitrary input or output ...
78 views

### Recursive Least squares (RLS) for mini batch

For my application I am considering a learning problem where I simulate a bunch of episodes say '$n$' first, and than carry out the recursive least squares update. Similar to $TD(1)$. I know that RLS ...
31 views

### Why are rows of Attention Weights in a Hopfield Transformer the same?

I'm working on building a Hopfield Transformer using the github code from the paper (https://github.com/ml-jku/hopfield-layers/tree/master/hflayers) to forecast a timeseries dataset with 48 variables, ...
382 views

### Why is there a Uniform and Normal version of He / Xavier initialization in DL libraries?

Two of the most popular initialization schemes for neural network weights today are Xavier and He. Both methods propose random weight initialization with a variance dependent on the number of input ...
46 views

### Can you illustrate how the weights in transformer model generated from a training sentence can be generalized to an unseen test sentence?

Can you show how the weights in transformer model are generalizable?
4k views

### Do neural network weights need to add up to one?

The idea that weights determine how much influence each input value from the current layer will have when calculating the input to the following layer reminds me of when my professors would say that ...
4k views

### Is there a proper initialization technique for the weight matrices in multi-head attention?

Self-attention layers have 4 learnable tensors (in the vanilla formulation): Query matrix $W_Q$ Key matrix $W_K$ Value matrix $W_V$ Output matrix $W_O$ Nice illustration from https://jalammar....
156 views

### What is actually being saved in the file when you save a model? For example a Tensorflow SavedModel file [closed]

I'm building a feature for my application that requires reading the properties of a saved ML model file (after it's trained). However, as I am pretty new to this field, I don't really understand the ...
60k views

### In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

My understanding is that the convolutional layer of a convolutional neural network has four dimensions: ...
103 views

### Does it make sense to store information in a variable defined inside a Pytorch nn.Module?

I have a pytorch model (custom model inherited from nn.Module). I'm developing some architecture, for which makes sense for my task to have a list defined in the model as: ...
3k views

### Why are the initial weights of neural networks randomly initialised?

This might sound silly to someone who has plenty of experience with neural networks but it bothers me... Random initial weights might give you better results that would be somewhat closer to what a ...
345 views

### How is InstructGPT a fine-tuned version of GPT-3 and at the same time has fewer parameters than the original GPT3?

I am reading the paper "Training language models to follow instructions with human feedback" It says: Our labelers provide demonstrations of the desired behavior on the input prompt ...
19 views

### Should we forget last weight update in neural network?

When training a neural network, the general process could be something like this: While error < min_error Forward pass Compute error and cost funcion Back propagation Update weights But when we ...
1 vote
18 views

### Giving Specified Data a Larger Value/Weight in a Model

I'm in the process of creating a model to classify an occupational code based on a job title & description. I have a large sample of labelled data to achieve this. The government has a resource ...
592 views

### In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?

Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
602 views

### Backpropagation of neural nets with shared weight

I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. ...
1 vote
136 views

### How to calculate the total number of inputs in CNN?

I search this kind of question for a while and I find many discussions involve on counting the number of parameters of a Convolutional Neural Network, but not on the inputs. Using the Fashion MNIST ...
93 views

### What are the applications in which the precision of the neural network's weights is unimportant?

While reading about Module in PyTorch, I came across a new data type called half datatype. half() method when calls on a Module ...
88 views

### Why the x-input should be multiplied by weight in an artificial neuron?

So why weight should be multiplied with input? Yes I know, weight is intended for tuning the connection strength of input that will affect output so that's will be useful for learning (CMIIW). But why ...
628 views

### Is there a way to freeze training for weights, but not biases in PyTorch? [closed]

I'm constructing a neural network where the weights of my first hidden layer (connected to the input) are all 1 (identity matrix), but the biases are variable. Is there a way to "freeze" any ...
1 vote
381 views

### Why should each filter have different weights for each input channel?

From the answers to this question In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?, I got the fact that ...
182 views

### Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
181 views

### Why and when do we need to normalize weights in Reinforcement Learning?

I recently came across this SO question, wherein the poster was asked to normalize their weights while using a function approximator with SARSA. I don't remember having to normalize any weights while ...
1 vote
41 views

### Do NNs suffer from lack of efficiency in network structure and suggesting training parameters?

I am working on dynamical systems using Optimal Control theory and trying to find the connection between this field and Machine Learning. Consider a simple 2-layer Neural Network (NN) where the ...
1 vote
36 views

### What do "large variables" and "small weights" mean in these sentences?

I'm trying to understand these two points from an article: Models with large variables i.e weight matrices. As a consequence such models have correspondingly large gradients and optimizer states. The ...
2k views

### Why do neural network weights have to be between 0 and 1?

I've been reading about neural networks for a long time, and I saw that in each one, the weights are always between 0 and 1. Why is this? I tried programming one, but the sigmoid function just seemed ...
596 views

### Why doesn't the high precision of neural network weights improve accuracy?

Consider the following paragraph from the subsubsection 3.5.2: A dtype for every occasion chapter named It starts with a tensor from the textbook titled Deep Learning with PyTorch by Eli Stevens et al....
72 views

### What are the numbers that are useful (may need to be stored) other than parameters of a model?

Consider the following method related to buffers in PyTorch ...
454 views

### Is there any advantage in viewing weights of a neural network as random variables?

In artificial intelligence, especially in machine learning, the inputs and outputs of neurons in a neural network can be viewed as random variables. And this view is highly useful in many ways. The ...
131 views

### How many layers and neurons in a FFNN do I need to make it equivalent to a CNN?

I started to learn machine learning early, and I studied the convolutional neural network and its ability to understand images and how it helps to reduce the number of parameters that need to be tuned....
195 views

### In TD(0) with linear function approximation, why is the gradient of $\hat v(S^{\prime}, \mathbf w)$ wrt parameters $\mathbf w$ not considered?

I am reading these slides. On page 38, the update for the parameters for the linear function approximation of TD(0) is given. I have a doubt regarding this. The cost function (RMSE) is given on page ...
1 vote
806 views

### Not able to understand Pytorch Tensor (Weight & Biases) Size for Linear Regression

Below are the two tensors ...
132 views

### How to train a neural network with few weights and biases held constant?

I am a beginner in neural networks. I am building a neural network with 3 layers. The input $X$ has 7 features and the output $Y$ is a real number. In the hidden layer, there are two nodes. The bottom ...
102 views

### Neural Nets: CNN confirming layer/filter arithmetic [duplicate]

I was hoping someone could just confirm some intuition about how convolutions work in convolutional neural networks. I have seen all of the tutorials on applying convolutional filters on an image, but ...
670 views

### What do the neural network's weights represent conceptually?

I understand how neural networks work and have studied their theory well. My question is: On the whole, is there a clear understanding of how mutation occurs within a neural network from the input ...
1 vote
14 views

### How does the distribution of the parameters change in logistic regression?

I have my own data to train a logistic regression model (for a multi-class classification task), and I want to know how the distribution of weight parameters changes after each update with gradient ...
1 vote
18 views

### Are there any recommendations on initialising a single parameter in deep learning?

I want to initialize a parameter, which is a single real number in my model. If you want the role of the parameter in the model, you can assume it as the parameter to multiply with the output of the ...
481 views

### What is the significance of weights in a feedforward neural network?

In a feedforward neural network, the inputs are fed directly to the outputs via a series of weights. What purpose do the weights serve, and how are they significant in this neural network?
284 views

### Are there neural networks with (hard) constraints on the weights?

I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For ...
304 views

### What does "adding class weights for an imbalanced dataset" mean in the case of multi-label classification?

Suppose I have the following toy data set: Each instance has multiple labels at a time. You can see I have 2 instances for Label2. However, only one instance for the other labels. It means that we ...
176 views

### model and trained model parameters on CIFAR-10 [closed]

I'm looking for different models (specifically ResNet18/20, ResNet32/34, VGG16, MobileNet and SqueezeNet) and their parameters after training (i.e., .pth file) that ...
I know they are not the same in working, but an input layer sends the input to $n$ neurons with a set of weights, based on these weights and the activation layer, it produces an output that can be fed ...