0
$\begingroup$

I am trying to understand how weights are actually gotten. What is generating them in a neural network? What is the algorithm that gives them certain values?

$\endgroup$
4
$\begingroup$

Typically, weights are randomly initialized. Then, as the model is optimized for its given task, those weights are steadily made "better" as determined by the network's loss function. This is also referred to as "training" the neural network.

By far the most popular way of updating weights in a neural net is the backpropagation algorithm, most simply with stochastic gradient descent (SGD). Essentially, the algorithm determines how much each individual weight contributed to the network's loss. It then updates that weight in the direction that would reduce the loss.

I recommend going through Michael Nielsen's online book to learn the basics.

$\endgroup$
0
$\begingroup$

I agree with @PhilipRaeisghasem, in most architectures, weights are initialized in a random manner. However, some research papers suggest applying a random normal distribution initialization to the weights in the case of Convolutional Neural Networks (for computer vision).

$\endgroup$
  • $\begingroup$ Absolutely. I didn't think getting into specific initialization distributions would be super beneficial for someone asking such a basic question. But you're right. $\endgroup$ – Philip Raeisghasem Mar 17 at 23:18
  • $\begingroup$ I absolutely overthought that question. Thanks for replying. $\endgroup$ – Justin Mar 18 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.