1
$\begingroup$

I am trying to understand how weights are actually gotten. What is generating them in a neural network? What is the algorithm that gives them certain values?

$\endgroup$
5
$\begingroup$

Typically, weights are randomly initialized. Then, as the model is optimized for its given task, those weights are steadily made "better" as determined by the network's loss function. This is also referred to as "training" the neural network.

By far the most popular way of updating weights in a neural net is the backpropagation algorithm, most simply with stochastic gradient descent (SGD). Essentially, the algorithm determines how much each individual weight contributed to the network's loss. It then updates that weight in the direction that would reduce the loss.

I recommend going through Michael Nielsen's online book to learn the basics.

| improve this answer | |
$\endgroup$
0
$\begingroup$

I agree with @PhilipRaeisghasem, in most architectures, weights are initialized in a random manner. However, some research papers suggest applying a random normal distribution initialization to the weights in the case of Convolutional Neural Networks (for computer vision).

| improve this answer | |
$\endgroup$
  • $\begingroup$ Absolutely. I didn't think getting into specific initialization distributions would be super beneficial for someone asking such a basic question. But you're right. $\endgroup$ – Philip Raeisghasem Mar 17 '19 at 23:18
  • $\begingroup$ I absolutely overthought that question. Thanks for replying. $\endgroup$ – iustin Mar 18 '19 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.