Documentation allows for
bias=True term, but they don't say where this parameter is actually stored. I wanted to clarify.
You would expect
nn.Linear(1, 1, bias=True) to have 1 free parameter (if you draw it out you have 1 input going to 1 output, so only 1 weight). But this isn't the case. An extra bias term is implicitly added to the model, to get 2 parameters. They probably should have said that, but this is how it works.
With this framework in mind, we get to my question:
Why does declaring
0 as the input size and
m as the output size initialize all the bias components to
0? It doesn't make sense why they would do this.
>>> for p in nn.Linear(0, 4, bias=True).parameters(): print(p) Parameter containing: tensor(, size=(4, 0), requires_grad=True) Parameter containing: tensor([0., 0., 0., 0.], requires_grad=True)