New answers tagged neural-networks
1
vote
What are the techniques used to initialize weights for neural networks?
Both Keras and PyTorch provide a range of initialization classes and functions. Of these, probably the two most commonly used are the Glorot (Xavier in Pytorch) and He (Kaiming in pytorch) ...
- 131
1
vote
Types of activation functions used in neural networks
Let me give a very easy explanation...
Activation Functions in neural networks are useful as they introduce non-linearity into the model, which helps in complex data pattern understanding.
Sigmoid ...
- 503
4
votes
Accepted
Which epoch is the best for me to choose?
The first question is not well defined. What does normal mean? It is clearly decreasing, so that is good. If you are asking if the fluctuations are normal, yes that is not uncommon. This is not an ...
- 2,300
1
vote
Is size of trained model on disk a good measure of model complexity?
The title of your question asks about model complexity. Yet the body of your question talks about this metric as being useful for embedded systems like a smartphone, which have more limited memory.
I ...
- 2,300
1
vote
Can I do incremental learning with different loss function in neural networks?
Yes this is possible. Many hyperparameters of training can be changed at any time when training a neural network model, and that can include the objective function. You can also change the dataset, ...
- 30.4k
2
votes
Accepted
In multilayer perceptron neural networks, the more complex a dataset is, the more difficult it will be to find adequate initial random weights?
but produce the same result. this is a fake news going around the ML community, they will have comparable performances, but the predictions won't be the same.
And no, initialization does not aim to ...
- 1,358
1
vote
How can I deal with random weights initialisation when predicting a time-series sine function?
From comments, you are testing your ideas with a very small neural networks.
The highly variable end result with large dependency on initial conditions is a common result of working with small numbers ...
- 30.4k
2
votes
Accepted
Why is it called multi-headed attention?
The original paper "Attention is all you need" mentions the following.
"
Instead of performing a single attention function with $ d_{model} $-dimensional keys, values and queries,
we ...
1
vote
Accepted
What is the potential issue of nested neural networks
Isn't this just a very short recurrent neural network? Same issues apply, although they are less severe since you aren't applying as many recurrent iterations. Once you start "nesting" them ...
- 737
0
votes
How to properly optimize shared network between actor and critic?
A common practice involves using a shared encoder, which is updated based solely on critic loss, as implemented in DrQv2.
- 51
8
votes
Accepted
Neural network for game
Neural networks do not directly take actions in games. Instead, some code needs to supply the current state of the game to the neural network, interpret its output and take the action. Typically yet ...
- 30.4k
3
votes
Does it make sense to apply batch normalization to a batch size of 1?
It's not possible to use batch normalization with a batch size of 1. Batch normalization requires you to calculate the variance of activation values in the current batch, and variance is undefined for ...
- 31
Top 50 recent answers are included