In machine learning, a tensor is a multi-dimensional array (i.e. a generalization of a matrix to more than 2 dimensions), which has some properties, such as the number of dimensions or the shape, and to which you can apply operations (for example, you can take the mean of all elements across all dimensions). So, a scalar is a 0-d tensor (no dimensions), a vector is a 1d tensor (1 dimension), a matrix a 2d tensor (2 dimensions), a cube a 3d tensor, and so on.
The name TensorFlow really comes from the fact that TensorFlow manipulates tensors all the time. The documentation of TensorFlow describes tensors as follows
Tensors are multi-dimensional arrays with a uniform type (called a dtype)
If you're familiar with NumPy, tensors are (kind of) like np.arrays.
In this context, you can think of the axis as an abstraction for dealing with or manipulating the dimensions (aka shape) of a tensor or to apply certain operations "across dimensions". It's not exactly a synonym for dimension because, in some software libraries (e.g. Keras and NumPy), you can also pass special arguments such as
-1 to the
axis parameter, which does not correspond to any dimension (there's no $-1$ dimension of a tensor). It's a way to apply certain operations to the tensor only across e.g. the first dimension (axis 0) or the second dimension (axis 2), and so on.
For example, let's say you have a matrix (2d tensor) $A$. You want to compute the average (aka mean) of the rows, i.e. sum all elements of row $i$, then divide by the number of elements (which corresponds to the number of columns in row $i$), and do this for all rows $i=1, \dots, K$. You can do this in NumPy by specifying that you want to apply the mean operation to
axis=1. So, after this operation, you will get a 1d tensor (a vector) with $K$ elements (i.e. the numbers of rows of the original matrix): this is what I mean by "across dimensions".
Here's a NumPy example that illustrates the concept.
import numpy as np
A = np.array([[1, 2, 3],
[1, 1, 1],
[0, 1, -1]])
# (1 + 2 + 3 + 1 + 1 + 1 + 0 + 1 + (-1)) / 9
m = np.mean(A)
print("Mean of all elements of A =", m)
# [(1 + 2 + 3)/3, (1 + 1 + 1)/3, (0 + 1 + (-1))/3]
m1 = np.mean(A, axis=1)
print("Mean of each row =", m1)
# [(1 + 1 + 0)/3, (2 + 1 + 1)/3, (3 + 1 + (-1))/3]
m2 = np.mean(A, axis=0)
print("Mean of each column =", m2)
Why would you want to compute (in this case) the mean of each row? Because, for example, each row $i$ may correspond to some data associated with a user $i$, so you may want to compute e.g. the average salary for each user for all months (in this case, there are only 3 columns, so 3 months).
Tensors did not originate in machine learning. In fact, in mathematics, tensors are well-known objects, which have some properties, which may be different than the properties associated with the tensors implemented in libraries like TensorFlow, which are really just multi-dimensional arrays with the necessary properties and methods for machine/deep learning. Tensors are also everywhere in quantum computing.
This answer that I wrote a while ago could also be useful.