Why does a zero-input network initialize all bias terms to 0? (pytorch)

Question

I'm following https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=nn%20linear#torch.nn.Linear

Documentation allows for bias=True term, but they don't say where this parameter is actually stored. I wanted to clarify.

You would expect nn.Linear(1, 1, bias=True) to have 1 free parameter (if you draw it out you have 1 input going to 1 output, so only 1 weight). But this isn't the case. An extra bias term is implicitly added to the model, to get 2 parameters. They probably should have said that, but this is how it works.

With this framework in mind, we get to my question:

Why does declaring 0 as the input size and m as the output size initialize all the bias components to 0? It doesn't make sense why they would do this.

For instance,

>>> for p in nn.Linear(0, 4, bias=True).parameters(): print(p)

Parameter containing:
tensor([], size=(4, 0), requires_grad=True)
Parameter containing:
tensor([0., 0., 0., 0.], requires_grad=True)

Answer 1

That's interesting. I'm guessing it has to do with the way PyTorch handles weight initialization. For linear layers, the bias range is set at the scale of $1/\sqrt{\text{in_features}}$ via the kaiming initialization function.

In your case, your input features is zero, so you get an undetermined scale. So I am guessing somewhere in the PyTorch code, this gets converted to or interpreted as zero (which is of course not mathematically correct but a common practice).