When describing tensors of higher order I feel like there is an overloading of the term dimension as it may be used to describe the order of the tensor but also the dimensionality of the... "orders"?

Assume one describes the third-order tensor produced by a convolutional layer and wants to refer to its width and height. Do you say spatial dimensions? Would you write about the channel dimension? Or rather the direction? Saying "spatial order" feels really weird. But staying with dimensions makes sentences like "The spatial dimensions are of equal dimensionality." (Disclaimer: Obviously you can avoid the issue here by restructuring, but doing this at every occasion does not feel like a satisfactory solution.).

  • 1
    $\begingroup$ just declare and continue in accordance. Most papers just say h,w dimension for example and stick to it $\endgroup$ – mshlis Jun 21 '19 at 14:19

Ambiguity in Terms

You are correct that there is something like overloading occurring in tensor terminology in posts and in software libraries. Confusing jargon often appears when those without the mathematical background use mathematical terms. You rarely find this confusion when reading NASA, Cambridge, MIT, or Cal-tech materials.

Tensors, Rank, Dimensions, and Channels

A tensor is a grouping of dimensions. The grouping typically represents a relation between quantities describing a system. The order (or rank) of the tensor is the number of dimensions in the grouping.

When this mathematical idea is applied formally to electrical engineering, chemical systems, biological systems, or computing systems, we can say a tensor is the grouping of the inputs and outputs of that system, abstracting the relation between them. The order (or rank) of the tensor is the total number of signal channels going to or from the system. The rank is simply the number of inputs plus the number of outputs.

That original conception was lost a little when the term tensor began to be applied to grouping inputs in one tensor and grouping outputs in another tensor, without any connection with their relationship, which is not formally correct. Those are simply vectors in mathematics, represented by arrays in computer programs.

A labelled data set prepared for machine learning is a discrete tensor formally, however it does not have a mathematical model applied to it yet. When convergence occurs in an artificial network, the result of training is a set of parameters that, if later applied to new inputs, will predict the outputs based on the net and its trained parameters. The configuration of the network cells had become the model and the trained network had become a tensor expression approximating the phenomenon that the training data represents.

The Specific Example in the Question

A third order tensor cannot have only two dimensions, width and height. It must have three dimensions, possibly width, height, and instantaneous value. Width and height are the independent variables, and instantaneous value is the dependent variable. So we have a rank of $1 + 2 = 3$.

There is no clear and formal definition of what dimensionality means, so it would be useful to discontinue its use until academic textbooks contain a consistent definition, but that's unlikely to occur. It will likely continue to be used by non-mathematicians somewhat arbitrarily. Therefore, stating that two tensors are of equal dimensionality remains ambiguous.

Domain and Range

The correct terms are domain and range. These terms unambiguously describe the variability of the independent and dependent variables of a relation respectively. In the case of an array of samples describing a tensor relationship between horizontal position, vertical position, and brightness, the horizontal domain corresponds to the width and the vertical domain corresponds to the height.

Consequently, one would not say, "Width and height are the spacial dimensions," or, "width and height are the channel dimensions."

It could unambiguously be stated, "Width in this article shall refer to the maximum horizontal pixel position minus its minimum plus one." Without saying something that complex, albeit accurate, this would be a clear statement.

The fourth convolution layer in the network has a domain of 1280, 768 and a range of IEEE 32 bit floats.

It is easily understood and close enough to technically correct to not cause readers with mathematical training to dismiss the writer as uneducated.

The equality question is now solved. The equality of domains could be stated clearly and unambiguously. Whether it sounds unusual is not particularly relevant.

The horizontal and vertical domains of network layers two and three are equal, both being [1280, 768].

(The reason [1280, 768] is often seen is because compilers of many languages, starting with FORTRAN or maybe even earlier, interpret that as a two dimensional domain for a two dimensional array. So that can be used among software people, although [768, 1280] is more likely used in programs since it is a video tradition to traverse left to right and then top to bottom.)

Misuse of the Term Tensor

It is incorrect to say that adding two arrays is an example of tensors manipulated in software. Tensors are more abstract than arrays. If one were to ...

  • Make symbolic manipulations to reduce a complex tensor equation to a simple one,
  • Show that two tensor expressions are equal for given domains, or
  • Determine the correlation of a set of discrete points to a model represented as a tensor equation,

... then one would be performing tensor mathematics.

There are many pieces of software that perform such symbolic manipulations, but assigning a tensor to an input sample in machine learning is not tensor math. Tensor object classes that are not part of a system of symbolic manipulation are simply an abstraction of the typical programming language's array structure, parameterizing rank in the constructor. Such a trivial abstraction is not nearly as deep as symbolic manipulation of tensor expressions.

| improve this answer | |
  • $\begingroup$ Thank you for this elaborate discussion! Despite lacking the mathematical background, I hope to now at least use the correct terminology in my future texts :) $\endgroup$ – weidler Jun 25 '19 at 12:11

By definition, tensors can be of any order (usually named differently if the order is less than three). So, I use $d_i$ to indicate the dimensionality of the $i$th facet.

Unless you have three or four-order tensors which each facet has a very specific meaning, naming the orders by terms such as special or time would be limiting.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.