# What is the definition of a trace of a tensor?

Tensor is a multi-dimensional ordered collection of elements, which is used in deep learning to store and process data as well as intermediate steps.

We are aware of the trace of a two-dimensional tensor i.e. matrix. It is defined as the sum of the diagonal elements of the matrix.

Is there any definition for the trace of a tensor?

The concepts of trace and tensor also appear in other contexts outside of machine learning (ML), like quantum computing, so an answer to your question may be given independently of ML, but that may not be useful, as these concepts may be defined and implemented differently in the context of ML, which seems to be the case.

The concept of trace, in mathematics, is apparently known as tensor contraction. I don't know if that definition is consistent with the definition(s)/implementation(s) of trace used in machine learning.

I found at least 2 (different) definitions of tensor trace in machine learning. The first definition is provided by the TensorFlow implementation of the trace of a tensor. For completeness, let me write here their definition of the trace.

trace(x) returns the sum along the main diagonal of each inner-most matrix in x. If x is of rank k with shape [I, J, K, ..., L, M, N], then output is a tensor of rank k-2 with dimensions [I, J, K, ..., L] where output[i, j, k, ..., l] = trace(x[i, j, k, ..., l, :, :]).

So, essentially, for each most inner 2d matrix in this tensor, you compute the trace (for that matrix), then return the result as another tensor, which has 2 fewer dimensions than the original tensor (because a matrix has 2 dimensions, and, by computing the tensor of a matrix, you reduce a matrix to a number, which is a 0-dimensional tensor).

They give these examples

x = tf.constant([[1, 2], [3, 4]])
tf.linalg.trace(x)  # 5

x = tf.constant([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
tf.linalg.trace(x)  # 15

x = tf.constant([[[1, 2, 3],
[4, 5, 6],
[7, 8, 9]],
[[-1, -2, -3],
[-4, -5, -6],
[-7, -8, -9]]])
tf.linalg.trace(x)  # [15, -15]


I think this definition is easy to understand, but I don't remember having ever used it, but I could be wrong.

In fact, PyTorch does not seem to implement the trace for tensors, but only matrices. If you executed the following code, you should get an error that tells you that's not possible.

import torch  # install also numpy

x = torch.tensor([[[1, 2, 3],
[4, 5, 6],
[7, 8, 9]],
[[-1, -2, -3],
[-4, -5, -6],
[-7, -8, -9]]])

print(torch.trace(x))  # [15, -15]? No, you get an error.


So, as I was suspecting, the tensor trace may not be terribly useful in ML, at least, for tensors with more than 2 dimensions.

By the way, in the paper A Survey on Tensor Techniques and Applications in Machine Learning (2019) Yuwang Ji et al. (p. 6 of the pdf), you find another definition of tensor trace, which I don't think it's equivalent to the definition used by the TF implementation.