I'm following https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=nn%20linear#torch.nn.Linear

Documentation allows for bias=True term, but they don't say where this parameter is actually stored. I wanted to clarify.

You would expect nn.Linear(1, 1, bias=True) to have 1 free parameter (if you draw it out you have 1 input going to 1 output, so only 1 weight). But this isn't the case. An extra bias term is implicitly added to the model, to get 2 parameters. They probably should have said that, but this is how it works.

With this framework in mind, we get to my question:

Why does declaring 0 as the input size and m as the output size initialize all the bias components to 0? It doesn't make sense why they would do this.

For instance,

>>> for p in nn.Linear(0, 4, bias=True).parameters(): print(p)

Parameter containing:
tensor([], size=(4, 0), requires_grad=True)
Parameter containing:
tensor([0., 0., 0., 0.], requires_grad=True)

  • 2
    $\begingroup$ I don't understand why you expect only one parameter in the first case with bias=True. You have activation = weight * input + bias. One input, one output, everything is scalar. The layer has two parameters: the weight, and the bias. $\endgroup$
    – maxy
    Oct 14, 2022 at 7:46

1 Answer 1


That's interesting. I'm guessing it has to do with the way PyTorch handles weight initialization. For linear layers, the bias range is set at the scale of $1/\sqrt{\text{in_features}}$ via the kaiming initialization function.

In your case, your input features is zero, so you get an undetermined scale. So I am guessing somewhere in the PyTorch code, this gets converted to or interpreted as zero (which is of course not mathematically correct but a common practice).


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .