How exactly is the dynamical unfolding implemented in ByteNet?

Question

I am thinking about making use of ByteNet (https://arxiv.org/abs/1610.10099) architecture for a project, and would like to get a better understanding of how the model works.

I've read through the paper about a million times, but can't figure out how exactly the dynamical unfolding is implemented.

I understand that the input sequence is mapped to a longer intermediate representation, whose length is a function of the input length, but don't understand how 1×1 convolutional layers are able to do this.

I am also unclear on how input sequences can be of variable length without any recurrence.

Any help would be appreciated! Or of anyone knows of any good YouTube videos that explain it, that would also be helpful.

Answer 1

It seems you main question is about the mechanism of variable input length handling of a series of 1d convolutional layers. This is mainly due to the mechanism of the 1d convolutional layer which avoids the limitations associated with RNNs like fixed-length inputs or the need for padding to a common length.

The encoder processes the source string into a representation and is formed of one-dimensional convolutional layers that use dilation but are not masked.

1d convolution aka cross correlation in ML is like dot-product operation between the 1d filter and the 1d input sequence in a consecutive sliding-window fashion. Often the input sequence will have multiple input channels aka feature dimensions and we can convolve each channel separately and add up the result using a different 1d filter for each input channel. This outputs a 1d vector of the same length as the input sequence which encodes the input assuming we ignore any potential boundary effect. Therefore in this way ByteNet's encoder can handle variable length sequence which is very similar to the usual 2d convolution for CNNs to be capable of handling variable length image input.

Finally the non-masked dilations between each 1d convolutional layer in the encoder network exponentially grows the region of the input that influences a neuron in a convolutional layer which is called receptive field, so the encoder can capture long range dependencies in a possibly long input sequence.