1
$\begingroup$

I am reading the paper Convolutional Sequence to Sequence Learning by Facebook AI researchers and having trouble to understand how the dimensions of convolutional filters work here. Please take a look at the relevant part of the paper below.

enter image description here

Let's say the input to the kernel X is k*d (say k=5 words of d=300 embedding dimenisonality). Therefore the input is 5*300. In a computer vision task a kernel would slide over parts of the image, in NLP you usually see kernel taking up the whole width of the input matrix. So I would expect kernel to be m*d (e.g. 3*300 - slide over 3 words and look at their whole embeddiings).

However, the kernel here is of dimensionality 2d x kd which in our hypothetical example would be 600*1500. I don't understand how this massive kernel would slide over an input that is by far lower dimensional (5*300). In computer vision you could zero-pad the input, but here zero-padding would basically turn the input matrix into mostly zeros with only a handful of meaningful numbers.

Thanks for shedding some light on it!

$\endgroup$
0
$\begingroup$

They are doing a matrix multiplication: consider $y = Ax, y \in \mathbb{R}^m, x \in \mathbb{R}^n, A \in M_\mathbb{R}(m,n)$. In the paper $x$ is a concatenation of $k$ elements of $\mathbb{R}^d$, so $x$ is long $kd$; $y$ is long $2d$.

$\endgroup$
0
$\begingroup$

First for clarity I want to differ "convolutional kernel" and "filter" here, let's say a filter has several convolutional kernels. Second in NLP 1D-convolution is mostly used, so different from CV, which typically use a $k \times k$ convolutional kernel, in NLP generally the kernel size is only $k$

Having listed above hope things could become a little clearer: If we adopt the same concepts as we seen in typical CV scenario, per my understanding, the size of convolutional kernel is only $k$, but the count of kernels, or the dimension of new "channel", is set to $2d$. One kernel maps $k \times d$ elements into one element, $2d$ kernels together convert the input to a vector of which the dimention is $2d$

As the calculation process is mapping a matrix $\mathbb{R}^{k \times d}$ to a vector $\mathbb{R}^{2d \times 1}$, if expressed in a form of matrix multiplication, or in a full-connected view, we can firstly expand the input matrix, concatenate all rows together to get a new big vector of which the dimension is $\mathbb{R}^{kd \times 1}$, then multiply a weight matrix $\boldsymbol{W} \in \mathbb{R}^{2d \times kd}$ could get the final vector

The key problem that roots the confusion is beause, in my opinion, the "kernel" imported here is a little different from the same concept used in a typical CNN. Again, if we describe in a traditional CNN terminology, the kernel size should be $k$, and there are $2d$ kernels applied.

Hope my understanding is correct.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.