What is meant by an axis of a tensor?

Question

Tensor is an ordered collection of elements. The elements are generally real numbers. Tensors are used in deep learning for storing data.

There is a wide usage of the word "axis" related to tensor. Axes are not the same as indices, which are used to access the elements of a tensor. An axis is not the same as an element of a tensor.

What exactly is an axis in a tensor? Is it also a (sub-)tensor obtained from the actual tensor? Or is it any other indexing mechanism? If yes, why it is used?

Suppose $a =[[1, 2, 3, 4],[5, 6, 7, 8],[9, 10, 11, 12],[13, 14, 15, 16]]$ is a tensor. Then what does axis do for $a$?

Answer 1

In machine learning, a tensor is a multi-dimensional array (i.e. a generalization of a matrix to more than 2 dimensions), which has some properties, such as the number of dimensions or the shape, and to which you can apply operations (for example, you can take the mean of all elements across all dimensions). So, a scalar is a 0-d tensor (no dimensions), a vector is a 1d tensor (1 dimension), a matrix a 2d tensor (2 dimensions), a cube a 3d tensor, and so on.

The name TensorFlow really comes from the fact that TensorFlow manipulates tensors all the time. The documentation of TensorFlow describes tensors as follows

Tensors are multi-dimensional arrays with a uniform type (called a dtype)

If you're familiar with NumPy, tensors are (kind of) like np.arrays.

In this context, you can think of the axis as an abstraction for dealing with or manipulating the dimensions (aka shape) of a tensor or to apply certain operations "across dimensions". It's not exactly a synonym for dimension because, in some software libraries (e.g. Keras and NumPy), you can also pass special arguments such as -1 to the axis parameter, which does not correspond to any dimension (there's no $-1$ dimension of a tensor). It's a way to apply certain operations to the tensor only across e.g. the first dimension (axis 0) or the second dimension (axis 2), and so on.

For example, let's say you have a matrix (2d tensor) $A$. You want to compute the average (aka mean) of the rows, i.e. sum all elements of row $i$, then divide by the number of elements (which corresponds to the number of columns in row $i$), and do this for all rows $i=1, \dots, K$. You can do this in NumPy by specifying that you want to apply the mean operation to axis=1. So, after this operation, you will get a 1d tensor (a vector) with $K$ elements (i.e. the numbers of rows of the original matrix): this is what I mean by "across dimensions".

Here's a NumPy example that illustrates the concept.

import numpy as np

A = np.array([[1, 2, 3],
              [1, 1, 1],
              [0, 1, -1]])

# (1 + 2 + 3 + 1 + 1 + 1 + 0 + 1 + (-1)) / 9
m = np.mean(A)
print("Mean of all elements of A =", m)

# [(1 + 2 + 3)/3, (1 + 1 + 1)/3, (0 + 1 + (-1))/3]
m1 = np.mean(A, axis=1)
print("Mean of each row =", m1)

# [(1 + 1 + 0)/3, (2 + 1 + 1)/3, (3 + 1 + (-1))/3]
m2 = np.mean(A, axis=0)
print("Mean of each column =", m2)

Why would you want to compute (in this case) the mean of each row? Because, for example, each row $i$ may correspond to some data associated with a user $i$, so you may want to compute e.g. the average salary for each user for all months (in this case, there are only 3 columns, so 3 months).

Tensors did not originate in machine learning. In fact, in mathematics, tensors are well-known objects, which have some properties, which may be different than the properties associated with the tensors implemented in libraries like TensorFlow, which are really just multi-dimensional arrays with the necessary properties and methods for machine/deep learning. Tensors are also everywhere in quantum computing.

This answer that I wrote a while ago could also be useful.

Answer 2

Imagine the tensor as a some generalized $n$-dimensional hyperrectangle sliced into $n$-dimensional hypercubes. Each element of the tensor is labeled by the position along the given axis, say $(x_1, x_2, \ldots)$.

Axis is not a property of tensor, rather the tensor is embedded in a $n$-dimensional space, where the axes are chosen along the sides of the hyperrectangle corresponding to the tensor.

There are many operations, that can be applied axiswise to tensor. Several examples:

• Mean along the axis (choose $0$ without loss of generality). Given the $n$-dimensional tensor $x_{i_1, i_2 \ldots i_n}$ , the result will be $n-1$-dimensional tensor $\frac{1}{N_0} \sum_{i_1} x_{i_1 i_2 \ldots i_n}$, where $N_0$ is the number of elements of the tensor along the $0$-th axis (height).
• Standard deviation along the axis. For each index $i_2 \ldots i_n$, calculate $\sqrt{\frac{1}{N_0} \sum_{i_1} (\bar{x}_{i_2 \ldots i_n} - x_{i_1 i_2 \ldots i_n})^2}$, where $\bar{x}$ is the mean from previous point, and the result will be again $n-1$-dimensional tensor.

For your example $a$ is a $2$-dimensional tensor with $2$ axes. $0$-th axis corresponds to rows, $1$-st axis corresponds to columns.