3
$\begingroup$

From the literature I have read so far, it is not clear how exactly the convolution operation is defined. It seems people use two different definitions:

Let us assume we are given an $n_w \times n_h \times d$ input tensor $I$ and an $m_w \times m_h \times d$ filter $F$ of $d$ kernels (I use the convention of referring to the depth-slices of filters as kernels. I also will call the depth slices of the input tensor channels). Let us also assume $F$ is the $j$th filter of $J$ filters.

Now to the definitions.

Option 1:

The convolution of $I$ with $F$ is obtained by sliding $F$ across $I$ and computing the Frobenius inner product between channel $k$ and kernerl $k$ at each position, adding the products and storing them in an output matrix. That matrix is the result of the convolution. It is also the $j$th feature map in the output tensor of the convolution layer.

Let $I \in \mathbb{R}^{n_w \times n_h \times d}$ and $F \in \mathbb{R}^{m_w \times m_h \times d}$. Let $s \in \mathbb{N}$ be the stride. The operation will only be defined if the smaller tensor fits within the larger tensor along its width and height a positive integer number of times when shifting by $s$, that is if and only if $k_w = (n_w - m_w) / s + 1\in \mathbb{N}$ and $k_h = (n_h - m_h) / s + 1\in \mathbb{N}$, where $k_w \times k_h \times d$ is the shape of the output tensor. Furthermore let ${f_x : i \mapsto (x - 1)s + i}$ be a function that returns the absolute index in the input tensor, given an index $x$ in the output tensor, the stride length $s$ and a relative index $i$. \begin{equation*} \begin{split} (I * F)_{x y} = & \sum_{k=1}^d \sum_{i = 1}^{m_w} \sum_{j = 1}^{m_h} I_{f_x(i) f_y(j) k} \cdot F_{i j k} \end{split} \end{equation*}

Option 2:

The convolutions (plural) of $I$ with $F$ are obtained by sliding $F$ across $I$ and computing the Frobenius inner product between channel $k$ and kernel $i$ at each position. Each product is stored in a matrix associated with the channel $k$. There is no adding of the products yet. The convolutions are the result matrices. The step where the matrices are added component wise to obtain the $j$th feature map of the output tensor of the convolution layer is not part of the convolution operation, but an independent step.

Let $I \in \mathbb{R}^{n_w \times n_h}$ and $F \in \mathbb{R}^{m_w \times m_h}$. Let $s \in \mathbb{N}$ be the stride. The operation will only be defined if the smaller matrix fits within the larger one along its width and height a positive integer number of times when shifting by $s$, that is, if and only if $k_w = (n_w - m_w) / s + 1\in \mathbb{N}$ and $k_h = (n_h - m_h) / s + 1\in \mathbb{N}$, where $k_w \times k_h$ is the shape of the output matrix. Furthermore let ${f_x : i \mapsto (x - 1)s + i}$ be a function that returns the absolute index in the input matrix, given an index $x$ in the output matrix, the stride length $s$ and a relative index $i$. \begin{equation*} \begin{split} (I * F)_{x y} = &\sum_{i = 1}^{m_w} \sum_{j = 1}^{m_h} I_{f_x(i) f_y(j)} \cdot F_{i j} \end{split} \end{equation*}

Which of these two definitions is the common one?

$\endgroup$
2
$\begingroup$

Both are incorrect.

using your notation
You do not take a sliding frobenius inner product of a singular channel of $I$ with $F$, but with all the channels at once. This may be easier to understand if you do not assume the number of channels of the input and output are the same (ie different number of input channels then filters). So lets say your input has $k_1$ channels and you have $k_2$ filters.

This means $shape(I) = (N, M, k_1)$ and each of the $k_2$ filters is of shape $(n, m,k_2)$ and your output shape is $(N-n+1, M-m+1, k_2)$ assuming your not using padding.

So i guess trying to put it in the manner you use: The convolution's output's $i^{th}$ channel is taking the sliding Frobenius inner product between all $n \times m$ cross-section of $I$ (including all input channels) with filter $F_i$.

$\endgroup$
  • 1
    $\begingroup$ @lotolmencre Oh I see, sorry for misunderstanding. I think this is a tricky one to answer, but I would say Option 1, as all instances of a convolutional layer I have seen have referred to at most the act of convolving and passing the outputs through an activation function as separate segments, nevermind the act of convolving (but not summing), then summing, then passing the outputs through an activation function. The reason I say it is hard to answer is I don't believe there is a formal definition (in terms of machine learning) $\endgroup$ – Recessive Jul 25 at 7:14
  • 1
    $\begingroup$ @lotolmencre From my understanding, the term "convolution" in regards to neural networks is a loose interpretation of the typical definition, which "is a mathematical operation on two functions (f and g) to produce a third function that expresses how the shape of one is modified by the other." I think this supports option 1, especially in the fact "The term convolution refers to both the result function and to the process of computing it." But even so, I can not answer this with 100% confidence, because I am unsure of a formal definition in regards to ML. $\endgroup$ – Recessive Jul 25 at 7:18
  • 1
    $\begingroup$ (sorry for late reply, was asleep) Its is the exact same definition just in the 2 dimensional sense of discrete convolution. Regarding the link you sent: Are you talking about the average pooling at the end? if so, that is an additional layer/op seperate form the convolution. $\endgroup$ – mshlis Jul 25 at 12:39
  • 1
    $\begingroup$ also note that after your edits: your definitions still are wrong, but the equation in option 1 is correct! $\endgroup$ – mshlis Jul 25 at 12:42
  • 1
    $\begingroup$ "between channel k and kernerl k at each position" $\endgroup$ – mshlis Jul 25 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.