2
$\begingroup$

I'm reading this paper: https://arxiv.org/pdf/1602.07576.pdf. I'll quote the relevant bits:

Deep neural networks produce a sequence of progressively more abstract representations by mapping the input through a series of parameterized functions. In the current generation of neural networks, the representation spaces are usually endowed with very minimal internal structure, such as that of a linear space $\mathbb{R}$^n.

In this paper we construct representations that have the structure of a linear $G$-space, for some chosen group $G$. This means that each vector in the representation space has a pose associated with it, which can be transformed by the elements of some group of transformations $G$. This additional structure allows us to model data more efficiently: A filter in a $G$-CNN detects co-occurrences of features that have the preferred relative pose [...]

A representation space can obtain its structure from other representation spaces to which it is connected. For this to work, the network or layer $\phi$ that maps one representation to another should be structure preserving. For $G$-spaces this means that $\phi$ has to be equivariant: $$\phi(T_gx)=T'_g\phi(x)$$That is, transforming an input $x$ by a transformation $g$ (forming $T_gx$) and then passing it through the learned map $\phi$ should give the same result as first mapping $x$ through $\phi$ and then transforming the representation.

Equivariance can be realized in many ways, and in particular the operators $T$ and $T'$ need not be the same. The only requirement for $T$ and $T'$ is that for any two transformations $g$ and $h$, we have $T(gh) = T (g)T (h)$ (i.e. $T$ is a linear representation of $G$).

I didn't understand the paragraph in bold. A structure preserving map is something that preserves some operation between elements in the underlying set. A simple example: if $f:\mathbb{R}^3\to\mathbb{R}$ such that $(x,y,z)^T\mapsto x+y+z$, then $$f(r+s)=f((r_1,r_2,r_3)^T+(s_1,s_2,s_3)^T)=f((r_1+s_1,r_2+s_2,r_3+s_3)) \\=r_1+s_1+r_2+s_2+r_3+s_3=f(r)+f(s)$$ where the addition in the far left term is in $\mathbb{R}^3$ and addition in the far right is in $\mathbb{R}$. So the map $f$ preserves the additional structure of addition.

In the quoted paragraph, $\phi$ is the structure preserving map, but what's the structure being preserved exactly? And why is the operator on the right different from one on the left? i.e. $T'$ on RHS instead of $T$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.