Is there any difference between affine transformation and linear transformation?

Question

Consider the following statements from A Simple Custom Module of PyTorch's documentation

To get started, let’s look at a simpler, custom version of PyTorch’s Linear module. This module applies an affine transformation to its input.

Since the paragraph is saying PyTorch’s Linear module, I am guessing that affine transformation is nothing but linear transformation.

Suppose $x = [x_1, x_2, x_3,\cdots,x_n]$ be an input, then the linear transformation on $x$ can be $a.x+b$, where $a$ and $b$ are $n-$ dimensional vectors of real numbers. And dot($.$) stands for dot product.

Is affine transformation same as the linear transformation? If yes, then why the name affine is used? Does it cover something more or less than linear transformation?

Answer 1

In linear algebra, a linear transformation (aka linear map or linear transform) $f: \mathcal{V} \rightarrow \mathcal{W}$ is a function that satisfies the following two conditions

1. $f(u + v)=f(u)+f(v)$ (additivity)
2. $f(\alpha u) = \alpha f(u)$ (scalar multiplication),

where

• $u$ and $v$ vectors (i.e. elements of a vector space, which can also be $\mathbb{R}$ [proof], some space of functions, etc.)
• $\alpha$ is a scalar (e.g. which can be a real number, but not necessarily)
• $\mathcal{V}$ and $\mathcal{W}$ are vector spaces (e.g. $\mathbb{R}$ or $\mathbb{R}^2$)

So, any function that satisfies these two conditions is a linear transformation.

In Euclidean geometry, $g(x) = ax + b$ is an affine transformation, which is generally not a linear transformation as defined in linear algebra. You can easily show that affine transformations are not linear transformations. For example, let $a = 1$ and $b = 2$, does $g$ satisfy the second condition above for any scalar $\alpha$? No. For example, let $\alpha = 3$, then $g(3x = y) = y + 2 = 3x + 2 \neq 3 g(x) = 3 (x + 2) = 3x + 6$.

However, in the context of neural networks, when people use the adjective "linear" they are often referring to a line. For example, in linear regression, you can have a bias (the $b$ in the affine transformation $g$ above), which would make the function not a linear transformation, but we still call it linear regression because we fit a line (hence the name linear regression) to the data.

So, no, an affine transformation is not a linear transformation as defined in linear algebra, but all linear transformations are affine. However, in machine learning, people often use the adjective linear to refer to straight-line models, which are generally represented by functions that are affine transformations. In this answer, I also talk about this issue.

Answer 2

The fact is you can always express an affine transformation as a linear transformation (more convenient because it is just a matrix/dot product).

For instance, given an input $\textbf{x}=[x_1, ..., x_n]$, some weights $\textbf{a} = [a_1, a_2, ..., a_n]$ and a bias $b \in \mathbb{R}$, you can express the affine operation $y = \textbf{a}\cdot \textbf{x} + b$ as :

$y = \tilde{\textbf{a}} \cdot \tilde{\textbf{x}}$, with $\tilde{\textbf{a}} = [a_1, ..., a_n, b]$ and $\tilde{\textbf{x}} = [x_1, ..., x_n, 1]$ (linear operation)

When your affine transformation is a function $f:\mathbb{R}^p \rightarrow \mathbb{R}^q$ with $\textbf{y}=f(\textbf{x})=A\textbf{x} + \textbf{b}$, you can use the same trick (by adding a column with the biases at the right end of the weight matrix $A$), so you get: $\textbf{y}=\tilde{A}\tilde{\textbf{x}}$

I found an example in this video, where Andrew Ng uses this trick for a simple Linear Regression.