How to visually or intuitively understand single element multi-dimensional tensors?

Question

Consider the following code in PyTorch

>>>torch.tensor([8]).shape
torch.Size([1])
>>>torch.tensor([[8]]).shape
torch.Size([1, 1])
>>>torch.tensor([[[8]]]).shape
torch.Size([1, 1, 1])   

We can notice that we want to store only a single element $8$ in a tensor. But it is possible in tensors to store $8$ in any n-dimensional tensor where $n \in \mathbb{N}$. In strict case $\mathbb{N}$ may be replaced by $\mathbb{W}$.

But, I am facing difficulty in understanding this fact of a single element contributing to all dimensions. If the element is present in all dimensions, then I am assuming that it has to be present multiple times, which is not the case. I can't understand how a single element is contributing any number of dimensions without repeating itself multiple times.

How to understand this phenomenon? How should I interpret or visualize this fact intuitively?

Answer 1

How should I interpret or visualize this fact intuitively?

You could visualize it as a point in a geometrical space:

• 8 is just a number
• [8] is just a number in a line
• [[8]] is a number in a plane
• [[[8]]] is a number in a space

The object (number 8) won't change. The space around it changes.

• You can never represent a complex object (3d-cube) in a simpler shape (2d-plane).
• But you can always represent a simpler object (2d-square) in a higher dimensional shape (3d-space).

A number is a simplest possible object, and therefore it "fits inside" (can be represented in) any dimension.

Answer 2

The number is not repeated if it is the only element of some high dimensional space.

An $n$-dimensional (vector) space is a set of objects (known as vectors, although this term is more commonly used to refer to the objects in $1$-dimensional spaces), which are ordered/organized/shaped in specific ways, but composed of scalars/numbers, and to which you can also apply certain operations.

It may be a good idea to think that the shape/organization of each object in an $n$-d space is determined by some "container", which has a certain shape, and that the container of the objects of higher-dimensional spaces generalizes (or it can include) the containers of lower-dimensional spaces.

• In the case of a $0$-dimensional space (e.g. $\mathbb{R}$), there's no container because you don't need it, as every object is composed of only one number (this is always the case!).

• In the case of $1$-dimensional spaces, an object could be composed of more than one number (but this does not have to be always the case: it depends on the specific $1$-d space, so there are many $1$-d spaces), so you need a way to organized these numbers. You organize them in a sequence that follows some direction. So, the objects $[0] \in \mathbb{R}^1$, $[1, 2] \in \mathbb{R}^2$, $[2, 3, 1] \in \mathbb{R}^3$, $[0, 4, 2, 2] \in \mathbb{R}^4$, etc., are all $1$-d objects because each of them is composed of one or more numbers, which are organized in sequence. $\mathbb{R}^1$, $\mathbb{R}^2$, $\mathbb{R}^3$, etc., are all $1$-d spaces, because they organize each of their objects in a sequence: their only difference is the number of numbers/scalars in each object.

• In the case of $2$-d spaces, you organize the objects not in a line, but in a rectangle. The rectangles can have different shapes, so they are not necessarily just squares. So, $[0] \in \mathbb{R}^{1 \times 1}$, $ \begin{bmatrix} 1 & 1\end{bmatrix} \in \mathbb{R}^{1 \times 2}$, $ \begin{bmatrix} 1 & 1 \\ 0 & 2\end{bmatrix} \in \mathbb{R}^{1 \times 2}$.

• In the case of $3$-d spaces, you organize each object into cuboids. This does not mean that these cuboids contain more than one number. In general, they could, but you also have cuboids that contain only one number, i.e. $[0] \in \mathbb{R}^{1 \times 1 \times 1}$, which, in PyTorch, would be printed as [[[0]]] to give you the idea that the number 0 is inside a container [0], which is inside another container [ ], which can contain only one small container that can contain only one number (e.g. [0], but you could also have had [1] or [10]), which is inside another container [ ], which can contain only one container, which can contain only another container, which, in turn, can contain only one number.

To give you an analogy, think of having many boxes of different sizes and also have balls that you can put inside the smallest boxes. These boxes are the "containers" and the balls are the numbers. You can put smaller boxes inside the bigger ones (such that the boxes cannot slide around) and you can put the balls inside only the smallest boxes (so that they do not move).