Questions tagged [weights]
For questions about the concept of weight (or parameter) of a machine learning model, such as a neural network or a linear regression model.
53
questions
0
votes
1answer
40 views
Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?
I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
0
votes
0answers
58 views
If I have a feature matrix of shape $1200 \times 2000$, should I have $2000$ input neurons?
I have an input feature matrix (row-normalized) of size $1200 \times 2000$.
When I am writing my neural network, is the number of input neurons equal to the number of columns, that is $2000$?
Is it ...
1
vote
1answer
27 views
Why are weights not initialized with mean=1?
I wonder why weights are initialized with zero-mean. It is one of the reasons, why deep architectures cannot be trained without skip connections. Without the skip connections, the zero initialization ...
1
vote
0answers
19 views
Which methods for weight initialization in Neural Networks are currently common practice?
I am currently researching the topic of weight initialization methods for (deep) neural networks and I'm a little stuck. The result of my work should be an overview of methods that are currently in ...
3
votes
1answer
93 views
Is the bias also a “weight” in a neural network?
I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a "weight" and a "...
0
votes
0answers
17 views
What is the effect and behavior of using mixed weight instead of normal weight matrix?
Suppose I try to find appropriate matrix A in differential equation $\dot{X}=A X$ using RNN.
Current state is $X=\begin{bmatrix}
x_{1}\\
x_{2}\\
\end{bmatrix}$, and desired trajectory state is $X_d=\...
0
votes
0answers
11 views
How to update all the weights in case only one data out of n signals is observable
If we have cost function as
$$E_i = (D_i -Y_i)^T Q (D_i -Y_i)$$,
where $$Q=\begin{bmatrix}
1 & 0 & 0\\
0 & 0 & 0\\
0 & 0 & 0
\end{bmatrix}$$( in case only one data signal can ...
2
votes
1answer
53 views
Is this LSTM layer learning anything?
I've trained a CNN-LSTM model but the results weren't satisfactory, so I took a look at my weight distributions and this is what I got:
I don't understand. Is this layer learning anything? Or no?
...
1
vote
1answer
36 views
How to train a neural network with few weights and biases held constant?
I am a beginner in neural networks. I am building a neural network with 3 layers. The input $X$ has 7 features and the output $Y$ is a real number. In the hidden layer, there are two nodes. The bottom ...
5
votes
1answer
134 views
Why did the developement of neural networks stop between 50s and 80s?
In a video lecture on the development of neural networks and the history of deep learning (you can start from minute 13), the lecturer (Yann LeCunn) said that the development of neural networks ...
2
votes
1answer
45 views
Does the weight vector form imply feature space curvature?
I came across this sentence when exploring a simple nearest neighbor classifier method using Euclidean distance (link):
The slightly odd thing about using the Euclidean distance to compare features ...
1
vote
1answer
107 views
Is there a convention on the order of multiplication of the weights with the inputs in neural nets?
Is there a convention on how the input data and the weights are multiplied? The input data can be anything, including the result from the previous layers.
There are two options:
Option 1:
$$\begin{...
3
votes
0answers
41 views
Why does sigmoid saturation prevent signal flow through the neuron?
As per these slides on page 35:
Sigmoids saturate and kill gradients.
when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero.
the gradient and ...
2
votes
0answers
47 views
Are there any new weight initialization techniques for DNN published after 2015?
Considering weights initialization in my personal projects, I always used some standard techniques such as:
Glorot (also known as Xavier) initialization (2010).
Mertens initialization (2010).
He ...
1
vote
1answer
49 views
What are acting as weights in a convolution neural network?
Looking at some old notes I took on CNN's and I wrote down that the weights in a CNN are acting like filters in a CNN but to be honest I don't really know what the weights are acting as in a CNN and ...
0
votes
0answers
22 views
How do we interpret the images of weights in logistic regression
The following images are
a) The weights of a logistic regression model trained on MNIST.
b) The sign of the weights of a logistic regression
How do these images represent the weights?
Would be ...
0
votes
1answer
31 views
Adding Class Weights for imbalanced dataset?
Suppose I have the following toy data set:
Each instance has multiple labels at a time. Now you can see I have 2 instances for Label2, however, only one, one instance for other labels. It means, we ...
4
votes
0answers
43 views
When is using weight regularization bad?
Regularization of weights (e.g. L1 or L2) keeps them small and standardized, which can help reduce data overfitting. From this article, regularization sounds favorable in many cases, but is it always ...
1
vote
0answers
60 views
Is the performance of a neural network, which was trained with encrypted data and weights, affected if the weights are decrypted?
Suppose that a neural network is trained with encrypted (for example, with homomorphic encryption and, more precisely, with the Paillier partial scheme) data. Moreover, suppose that it is also trained ...
3
votes
0answers
58 views
Why is there a Uniform and Normal version of He / Xavier initialization in DL libraries?
Two of the most popular initialization schemes for neural network weights today are Xavier and He. Both methods propose random weight initialization with a variance dependent on the number of input ...
1
vote
1answer
33 views
In attention models with multiple layers, are weight matrices shared across layers?
In articles that describe neural architectures with multiple attention layers of the same form, are the weight matrices usually the same across the layers? Consider as an example, "Attention is ...
1
vote
2answers
91 views
In the multi-head attention mechanism of the transformer, why do we need both $W_i^Q$ and ${W_i^K}^T$?
In the Attention is all you need paper, on the 4th page, we have equation 1, which describes the self-attention mechanism of the transformer architecture
$$
\text { Attention }(Q, K, V)=\operatorname{...
1
vote
0answers
41 views
Why should variance(output) equal variance(input) in Xavier Initialisation?
In a lot of explanations online for Xavier Initialization, I see the following:
With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
0
votes
0answers
41 views
How many parameters does the AI of a typical self driving car have?
I can find the number of parameters AIs such as GPT3 or OpenAI Five have, but I'm having a difficult time googling the number of parameters a self driving car has because Google searches keeps using ...
2
votes
0answers
43 views
How efficient is SCAWI weight initialization method?
I'm currently in the middle of a project (for my thesis) constructing a deep neural network. Since I'm still in the research part, I'm trying to find various ways and techniques to initialize weights. ...
1
vote
1answer
48 views
Should the range and initial values of weights and biases be adjusted to fit input and output data?
As a routine (in typical everyday tasks) of a data scientist, should they usually decide about weights and biases range and initial values as a function of which data they are planning to insert as ...
2
votes
1answer
95 views
Which hyperparameters in neural network are accesible to users adjustment
I am new to Neural Networks and my questions are still very basic.
I know that most of neural networks allow and even ask user to chose hyper-parameters like:
amount of hidden layers
amount of ...
4
votes
2answers
117 views
What is the goal of weight initialization in neural networks?
This is a simple question. I know the weights in a neural network can be initialized in many different ways like: random uniform distribution, normal distribution, and Xavier initialization. But what ...
1
vote
1answer
70 views
What are some examples of functions that machine learning models compute?
My simple understanding of AI is that it is based on a mathematical model of a problem. If I understood correctly, the model is a polynomial equation and its weights are calculated by training the ...
1
vote
1answer
39 views
Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?
I'm looking at some baseline implementations of RL agents on the Pendulum environment. My guess was to use a relatively small neural net (~100 parameters).
I'm comparing my solution with some ...
2
votes
1answer
48 views
When would bias regularisation and activation regularisation be necessary?
For Keras on TensorFlow, a layer class constructor comes with these:
kernel_regularizer=...
bias_regularizer=...
...
2
votes
1answer
51 views
Does the number of parameters in a convolutional neuronal network increase if the input dimension increases?
If I have a convolutional neuronal network, does the input dimension change the number of parameters? And if yes, why? If the sizes and lengths of the filters are still the same, how can the number of ...
1
vote
0answers
61 views
LSTM - MAPE Loss Function gives Better Results when Data is De-Scaled before Loss Calculation
I am building an LSTM for predicting a price chart. MAPE resulted in the best loss function compared to ...
2
votes
1answer
57 views
Is better to spend parameters on weights or bias?
If a neural network has a limited number of neuron parameters to find, -let's say only 1000 parameters-, it is generally better to spend the parameters on weights or neuron bias?
For example, if each ...
2
votes
3answers
187 views
Backpropagation of neural nets with shared weight
I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. ...
1
vote
2answers
115 views
How are non-linear surfaces formed in the training of a neural network?
Desperate trying to understand something for couple of weeks. All those questions are actually one big question.Please help me. Time-codes and screens in my question refer to this great(IMHO) 3d ...
1
vote
0answers
38 views
Is having binary randomized unchanging neural network weights a good idea?
I am creating a neural network to experiment with, and I was wondering:
If I have weights randomly initialized to be either 1 or 0 for each neuron, and then I made it so that the weights cannot be ...
2
votes
1answer
52 views
How are weights for weighted x-entropy loss on imbalanced data calculated?
I am trying to build a classifier which should be trained with the cross entropy loss. The training data is highly class-imbalanced. To tackle this, I've gone through the advice of the tensorflow docs
...
4
votes
2answers
179 views
Can neurons in MLP and filters in CNN be compared?
I know they are not the same in working, but an input layer sends the input to $n$ neurons with a set of weights, based on these weights and the activation layer, it produces an output that can be fed ...
1
vote
1answer
33 views
Why I have a different number of terms in word2vec and TFIDF? How I can fix it?
I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. I am using the same corpus for ...
1
vote
0answers
34 views
Function to update weights in back-propagation
I am trying to wrap my head around how weights get updated during back propagation. I've been going through a school book and I have the following setup for an ANN with 1 hidden layer, a couple of ...
3
votes
1answer
53 views
How are newer weight initialization techniques better than zero or random initialization?
How do newer weight initialization techniques (He, Xavier, etc) improve results over zero or random initialization of weights in a neural network? Is there any mathematical evidence behind this?
1
vote
1answer
68 views
Param count in last layer high, how can I decrease?
Not sure where to put this... I am trying to create a convolutional architecture for a DQN in keras, and I want to know why my param count is so high for my last layer compared to the rest of the ...
1
vote
0answers
37 views
Interpretability of feature weights from Gaussian process classifier
Suppose I trained a Gaussian process classifier with a linear kernel (using GPML toolbox) and got some feature weights for each input feature.
My question is then:
Does it/When does it make sense ...
3
votes
1answer
193 views
How does the neural-network know how to tweak weights for a specific neuron?
I know backpropagation uses cost and gradient descent to tweak the weights to minimize the cost. But how does it know which weights to give more weight to in the first place? Is there something inside ...
1
vote
1answer
49 views
Classification with deeplearning : clean start vs continue training
I trained some weights to identify apples and oranges (using YOLOv3).
If I want to be able to identify peaches, which approach is usually recommended:
Start clean and train the 3 classes.
Train ...
1
vote
1answer
53 views
How to plot Loss Landscape with more than 2 weights in the network
For a single neuron with 2 weights, I can plot the loss landscape and it looks like this (OR data, sigmoid activation, MAE loss):
But, when the neuron accepts more inputs, which means more than 2 ...
3
votes
1answer
84 views
How are the parameters of the Bernoulli distribution learned?
In the paper Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask, they learn a mask for the network by setting up the mask parameters as $M_i = Bern(\sigma(v_i))$. Where $M$ is the ...
1
vote
1answer
177 views
Why should each filter have different weights for each input channel?
From the answers to this question In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?, I got the fact that ...
1
vote
1answer
55 views
How many parameter would there be in a logistic regression model used to classify reviews into “good” or “bad”?
Suppose we want to classify a review as good ($1$) or bad ($0$). We have a training data set of $10,000$ reviews. Also, suppose we have a vocabulary of $100,000$ words $w_1, \dots, w_{100,000}$. So ...
