Is adding the Frobenius inner products between filter and input part of convolution or a separate step?

Question

From the literature I have read so far, it is not clear how exactly the convolution operation is defined. It seems people use two different definitions:

Let us assume we are given an $n_w \times n_h \times d$ input tensor $I$ and an $m_w \times m_h \times d$ filter $F$ of $d$ kernels (I use the convention of referring to the depth-slices of filters as kernels. I also will call the depth slices of the input tensor channels). Let us also assume $F$ is the $j$th filter of $J$ filters.

Now to the definitions.

Option 1:

The convolution of $I$ with $F$ is obtained by sliding $F$ across $I$ and computing the Frobenius inner product between channel $k$ and kernel $k$ at each position, adding the products and storing them in an output matrix. That matrix is the result of the convolution. It is also the $j$th feature map in the output tensor of the convolution layer.

Let $I \in \mathbb{R}^{n_w \times n_h \times d}$ and $F \in \mathbb{R}^{m_w \times m_h \times d}$. Let $s \in \mathbb{N}$ be the stride. The operation will only be defined if the smaller tensor fits within the larger tensor along its width and height a positive integer number of times when shifting by $s$, that is if and only if $k_w = (n_w - m_w) / s + 1\in \mathbb{N}$ and $k_h = (n_h - m_h) / s + 1\in \mathbb{N}$, where $k_w \times k_h \times d$ is the shape of the output tensor. Furthermore let ${f_x : i \mapsto (x - 1)s + i}$ be a function that returns the absolute index in the input tensor, given an index $x$ in the output tensor, the stride length $s$ and a relative index $i$. \begin{equation*} \begin{split} (I * F)_{x y} = & \sum_{k=1}^d \sum_{i = 1}^{m_w} \sum_{j = 1}^{m_h} I_{f_x(i) f_y(j) k} \cdot F_{i j k} \end{split} \end{equation*}

Option 2:

The convolutions (plural) of $I$ with $F$ are obtained by sliding $F$ across $I$ and computing the Frobenius inner product between channel $k$ and kernel $i$ at each position. Each product is stored in a matrix associated with the channel $k$. There is no adding of the products yet. The convolutions are the result matrices. The step where the matrices are added component wise to obtain the $j$th feature map of the output tensor of the convolution layer is not part of the convolution operation, but an independent step.

Let $I \in \mathbb{R}^{n_w \times n_h}$ and $F \in \mathbb{R}^{m_w \times m_h}$. Let $s \in \mathbb{N}$ be the stride. The operation will only be defined if the smaller matrix fits within the larger one along its width and height a positive integer number of times when shifting by $s$, that is, if and only if $k_w = (n_w - m_w) / s + 1\in \mathbb{N}$ and $k_h = (n_h - m_h) / s + 1\in \mathbb{N}$, where $k_w \times k_h$ is the shape of the output matrix. Furthermore let ${f_x : i \mapsto (x - 1)s + i}$ be a function that returns the absolute index in the input matrix, given an index $x$ in the output matrix, the stride length $s$ and a relative index $i$. \begin{equation*} \begin{split} (I * F)_{x y} = &\sum_{i = 1}^{m_w} \sum_{j = 1}^{m_h} I_{f_x(i) f_y(j)} \cdot F_{i j} \end{split} \end{equation*}

Which of these two definitions is the common one?

Answer 1

Both are incorrect.

using your notation
You do not take a sliding frobenius inner product of a singular channel of $I$ with $F$, but with all the channels at once. This may be easier to understand if you do not assume the number of channels of the input and output are the same (ie different number of input channels then filters). So lets say your input has $k_1$ channels and you have $k_2$ filters.

This means $shape(I) = (N, M, k_1)$ and each of the $k_2$ filters is of shape $(n, m,k_2)$ and your output shape is $(N-n+1, M-m+1, k_2)$ assuming your not using padding.

So i guess trying to put it in the manner you use: The convolution's output's $i^{th}$ channel is taking the sliding Frobenius inner product between all $n \times m$ cross-section of $I$ (including all input channels) with filter $F_i$.