Where do 'random seeds' get used in deep neural networks?

Question

I know that when creating neural networks it's standard practice to create a 'random seed' so that you can get producible results in your models. I have a couple of questions regarding this:

• Is the seed just something that is used in the 'learning' phase of the network or does it get saved? i.e. is it saved into the model itself and used by others if they decide to implement a model you created?
• Does it matter what you choose to be the seed? Should the number have a certain length?
• At what step of the creation of a model does this seed get used and how does it get used?

Other information about 'random seeds' would be welcomed! But these are my general questions.

Answer 1

I suppose the most common part where it will be used is in the initialization of weights before training; the best ways currently known to do that involve randomness.

If you use Dropout during training (randomly setting some activation levels to zero to combat overfitting), that also involves randomness, so your seed could also have influence there. Dropout should not be used anymore after training, although it could if you accidentally implement it to be used there. If you don't make that mistake though, your seed here should also only matter during training.

Depending on implementation, I suppose the seed could also have influence on random ordering of input data between epochs / random selection of minibatches from the training dataset during training. Of course, this is very much implementation-dependent. If you end up implement that kind of data processing yourself and don't do it through some framework, you'll also be the one who determines what random seed has influence on that process.

In general (barring any special cases that I'm unaware of), a Neural Network should behave deterministically after training; if you give it the same input, it should provide the same output, your random seed should no longer have influence after training.