What is the stride information of an image referring here?

Question

In convolutional neural networks, the convolution and pooling operations have a parameter known as stride, which decides the amount of jump the kernel needs to do on the input image. You can get more information regarding stride from follows taken from here

Stride is the number of pixels shifts over the input matrix. When the stride is 1 then we move the filters to 1 pixel at a time. When the stride is 2 then we move the filters to 2 pixels at a time and so on.

But, I am not getting what does it mean by stride information of an image at the tensor level. Consider the following paragraph from the chapter named Real-world data representation using tensors from the textbook titled Deep Learning with PyTorch by Eli Stevens et al.

img = torch.from_numpy(img_arr)
out = img.permute(2, 0, 1)

We’ve seen this previously, but note that this operation does not make a copy of the tensor data. Instead, out uses the same underlying storage as img and only plays with the size and stride information at the tensor level. This is convenient because the operation is very cheap; but just as a heads-up: changing a pixel in img will lead to a change in out.

It is mentioning about the stride information at the tensor level of an image. Do they mean the strides that are related to the CNN, pooling, etc., or are they referring to any other stride information?

Answer 1

Do they mean the strides that are related to the CNN, pooling, etc., or are they referring to any other stride information?

The stride referred to by the quote "only plays with the size and stride information at the tensor level" is referring to internal storage of tensors. Luckily in most normal conversations about AI logic you do not care about this, it has nothing to do with neural network implementation, but is a lower-level optimisation (compared to some other ways of implementing tensors that would not use an internal stride).

Most general-purpose computer chips do not use multi-part addressing for data, such as (x,y,z), but instead use a single memory location number. To construct a data structure to represent a 2-or-more dimensional tensor means combining more fundamental representations. There are a few different ways to do this. One might be an array of arrays, where the outer array contains pointers to memory addresses of inner arrays. However, a single array with some management numbers attached to it (typically at the beginning, or even as a separate data structure, so you can have multiple tensor views of the same data) has some advantages - some operations on the array are far more efficient. Reshaping and transposing are two operations that can be implemented far more efficiently when the storage is arranged in this way.

So, along with the tensor cell data (the number inside each location), a typical tensor implementation will store information separately about how that data is structured. One useful piece of data is the stride, which is how many memory locations to add for each step in a particular direction in the n-indices view of the data. For example in a 2-indexed tensor with shape (6,8) where the data is stored in rows, then incrementing the column index will take a +1 step in memory location, whilst incrementing the row index will take a +8 step (the number of columns in a row). This is the row stride for that shape.

Technically these two types of stride are conceptually very similar. The difference is the level within the code that they are implemented in. Usually for most work using a tensor library you only care about this "internal" stride happening because it gives you confidence that there is a low cost to a particular tensor operation, such as generating a different shaped view.

Most importantly, unless you are working on your own tensor library, or digging into the internals of one using a low-level language (C), then you do not need to specify the stride values used. Instead you can take the quoted text as a declaration that a particular operation is implemented efficiently and you can use it in your own code with confidence that there is not a high CPU (or GPU) cost to using it.