1
$\begingroup$

The basic layers for performing convolution operation in PyTorch are

nn.Conv1d:   Applies a 1D convolution over an input signal composed of several input planes.

nn.Conv2d:   Applies a 2D convolution over an input signal composed of several input planes.

nn.Conv3d:   Applies a 3D convolution over an input signal composed of several input planes.

Along with them, there are lazy versions to each of the aforementioned layers. They are

nn.LazyConv1d:    A torch.nn.Conv1d module with lazy initialization of the in_channels argument of the Conv1d that is inferred from the input.size(1).

nn.LazyConv2d:    A torch.nn.Conv2d module with lazy initialization of the in_channels argument of the Conv2d that is inferred from the input.size(1).

nn.LazyConv3d:    A torch.nn.Conv3d module with lazy initialization of the in_channels argument of the Conv3d that is inferred from the input.size(1).

We can observe from the description of lazy layers that the in_channels argument undergoes lazy initalization. Along with it, the attributes that will be lazily initialized are weight and bias.

Lazy Initialization is a performance optimization where you defer (potentially expensive) object creation until just before you actually need it. Lazy initialization is primarily used to improve performance, avoid wasteful computation, and reduce program memory requirements

Since in_channels, weight and bias are undergoing lazy initialization in lazy convolution layers of PyTorch, I am guessing that there may be cases that the layers can perform convolution operation without the need of in_channels, weight and bias or bypassing some of them.

Am I guessing correct? Are there any cases in which convolution operation is said to be done without initializing weights or number of input channels? If no, what is the gain we are getting by make such lazy initialization?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.