0
$\begingroup$

I know that neural networks don't need to have exactly the same weights, and that different networks trained on the same problem may result in different final weights, but produce comparable results. And i know that the initial weights initialized randomly are used to help the optimization procedures.

In multilayer perceptron neural networks, during training, using a dataset with a simpler pattern such as Iris, the network is more likely to find adequate initial weights using random weight initialization(perhaps because the pattern is simpler, and could there are more possibilities of initial weights that would be suitable?

And for example, if I use a much more complex dataset, which has a much more challenging pattern for the model. During random initial weight initialization, would it be much more difficult to find suitable initial weights so that the network can perform well?

In other words, the more complex a dataset is, the more difficult it will be to find adequate initial random weights?

Detail: the neural network I'm using, to start the random weights, generates pseudo-random numbers (between 0 and 1), using the same seed(42) to guarantee reproducibility.

$\endgroup$

1 Answer 1

2
$\begingroup$

but produce the same result. this is a fake news going around the ML community, they will have comparable performances, but the predictions won't be the same.

And no, initialization does not aim to get good initial performances, but to help the optimization procedures (so avoiding dead gradients, smooth locally the loss landscape and so on)

So, to answer to your question: _ the more complex a dataset is, the more difficult it will be to find adequate initial random weights_ no, it's pretty much impossible to find a random initial guess for neural network that works even for the simplest of the task, mainly due to stacked non linearities

The only initialization that is based on good guesses is called pretraining, but it's far from being random

$\endgroup$
5
  • $\begingroup$ Thanks for clarifying some points! But I have a question: I know that the initial weights initialized randomly are to help the optimization procedures (so avoiding dead gradients, smooth locally the loss landscape and so on) as you said. However, if this initialization is not good, would this hinder the training process? $\endgroup$
    – will The J
    Nov 19, 2023 at 18:56
  • 1
    $\begingroup$ @willTheJ yes, suppose to have normalized your feature in 0-1 (so they are all positive), and you initialize your weights with negative numbers, then $relu(Wx+b)$ will always be 0, thus no learning will happen at all. So yes, initialization is "an art", because it can most likely hinder your training (mainly the initial part), however now we have good guesses for most of the known activation functions $\endgroup$
    – Alberto
    Nov 19, 2023 at 19:48
  • $\begingroup$ Thanks for ReLu's example!. For example, the ReLu would produce output 0 if the value is negative, but what if for example it was the Sigmoid, would no learning occur either? $\endgroup$
    – will The J
    Nov 19, 2023 at 19:55
  • 1
    $\begingroup$ @willTheJ you can still have problems, for example initializing with big values your parameters $W,b$, doing $\sigma(Wx+b)$ will lead you to the tails of the sigmoid function, which has pretty much 0 gradient, so the learning will be very very slow $\endgroup$
    – Alberto
    Nov 19, 2023 at 23:12
  • $\begingroup$ Thanks for explainations! $\endgroup$
    – will The J
    Nov 20, 2023 at 13:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .