How can I learn tensors for deep learning?

Question

I've seen in most deep learning papers use tensors. I understood what tensors are, but I want to dive into them, because I think that might be beneficial for further studies in Artificial Intelligence. Do you have any suggestion (e.g. books or papers) about that?

Answer 1

In deep learning (and, in general, machine learning), tensors are multi-dimensional arrays. You can perform some operations on these multi-dimensional arrays (depending also on the specific implementations and libraries). These operations are similar to the operations you can apply to vectors or matrices, which are just specific examples of multi-dimensional arrays. Examples of these operations are

• indexing and slicing (if you are familiar with Python, these terms should not scare you)
• algebraic operations (such as multiplication of a tensor with another tensor, which includes numbers, vectors or matrices), which typically support broadcasting
• reshaping (i.e. change the shape of the tensor)
• conversion to or from another format (e.g. a string)

TensorFlow provides an article that discusses these tensors, so I suggest that you read it.

In mathematics, tensors are not just multi-dimensional arrays. They are multi-dimensional arrays that need to satisfy certain properties (in the same way that matrices need to satisfy certain properties to be called matrices) and are equipped with certain operations.

The paper A Survey on Tensor Techniques and Applications in Machine Learning (2019) Yuwang Ji et al., published by IEEE, provides a comprehensive overview of tensors in mathematics. It is full of diagrams that illustrate the concepts and the explanations are concise. Some of the explanations in this paper may not be very useful to develop deep learning applications, but some of the illustrations and explanations (especially, in the first pages, which are the only ones that I read) will give you some intuition behind the tensors or multi-dimensional arrays used in deep learning).

So, tensors in deep learning (DL) may not be exactly equivalent to the tensor objects in mathematics, because they may not satisfy all the required properties or some of the operations in the specific libraries may not be implemented, but it is fine to call them tensors because a tensor in mathematics is also a multi-dimensional array, to which you can apply operations (some of them are implemented in the DL libraries).