# 1D Sequence Classification with self-supervised learning

I am working on a multi-class classification task on long one-dimensional sequences. The sequence length may vary in the range $$[512, 30720]$$, and there is one feature associated each time-step in the range. This means that the input to the model is of the shape $$(N, 1, L)$$ where $$N$$ and $$L$$ are the variables for the batch size and sequence length respectively. The singleton feature dimension contains values in the range $$[0, 40]$$. A small slice of a sequence might look like this: $$[0,3,2,1,2,7,3]$$.

The standard way of using deep learning (i.e. supervised learning) to solve this task is by choosing some neural network architecture (our model) with an appropriate inductive bias towards the data and encode the features, propagate it threw the neural network, optional pooling, decode the features and finally compute the loss w.r.t the ground truth in that order. In PyTorch this process would look something like this (the model is a basic ResNet block):

import torch

input_shape = (5, 1, 1024) # batch size, features, timesteps
X = torch.randint(0, 14, input_shape).type(torch.float) # sample input
y_true = torch.tensor([0, 0, 1, 2, 1]) # sample ground-truth (3 classes)

d_model = 64 # 64 is our hidden dimension
encoder = torch.nn.Conv1d(in_channels=1, out_channels=d_model, kernel_size=1)

class Residual(torch.nn.Module):
def __init__(self, m):
super().__init__()
self.m = m

def forward(self, x):
return self.m(x) + x

model = torch.nn.Sequential(
Residual(
torch.nn.Sequential(
torch.nn.BatchNorm1d(d_model),
torch.nn.ReLU(),
torch.nn.BatchNorm1d(d_model),
),
),
torch.nn.ReLU(),
)

pooler = lambda x: x.mean(2) # average pooling over the length dimension
decoder = torch.nn.Linear(d_model, 3) # 3 output logits for each class

criterion = torch.nn.CrossEntropyLoss()

# training pipeline
X = encoder(X) # (N, 1, L) -> (N, H, L)
X = model(X) # (N, H, L) -> (N, H, L)
X = pooler(X) # (N, H, L) -> (N, H)
y_pred = decoder(X) # (N, H) -> (N, 3)
loss = criterion(y_pred, y_true)
loss.backward()


Assuming that the neural network architecture model is appropriate, I have considered using self-supervised learning (SSL) to boost the final classification accuracy on the withheld test set. I have seen the method gain attention in language modelling tasks with Transformers (i.e. BERT), but how does one apply SSL in the general case outside the language task domain? Is this approach successful without using Transformers?

As I understand, SSL is used on unlabeled data to learn the underlying structure of the data and is followed by a fine-tuning stage - i.e. supervised learning with the pretrained weights. Two SSL tasks that I can think of are 1) masking out certain timesteps and letting the model predict the values of the missing timesteps, and 2) training the model to predict the value of the next timestep given the previous timesteps. I'll use the following image to illustrate:

Intuitively, I think this should work but I am not sure how to implement it in a practical setting with PyTorch. For instance, when predicting the next timestep (red bar) can I treat it as a classification task with (40) classes? Would it work for varying sequence lengths? How big proportion of the training data should be used for pretraining?

• Can you please put your specific main question in the title? "1D Sequence Classification with self-supervised learning" is not really a question.
– nbro
Apr 20, 2022 at 12:42

In SSL (language modelling, for example), you do not have any explicit labels, just sequences of words that make sense together. SSL tries to model the language by next-word prediction, but the words it predicts are also parts of the sequence(s). The main goal of language modelling is to create (contextual) representations/embeddings for the words in the language, such that when put together, they preserve as much of the "meaning". Having modeled the representations, then you can change the head (last few layers) and fine-tune it on your classification/prediction task. This approach is important and useful when you want to learn the representations/features of the tokens in the sequence.

How does one apply SSL in the general case [...]?

The better question is, when does one apply SSL? I think it's usually when you want to learning some representations on unlabeled data, followed by the fine-tuning stage, on a downstream task.

I would say SSL is not appropriate in this context. Your data consists of the sequences (features) and the ground truths (classification labels). As far as I understand, there is no unlabeled data that you can use in the SSL way in this context. You don't need to know how to predict values in the sequence (missing/next), because you always know the representations of the features in your sequence (those singleton values).

The difference between this and language modeling is that in LM you do not have fixed, existing numeric values for each word, so you also want to learn these representations (word emebddings are not trivial; What is the best way to obtain them?).

• Thanks, I believe that all of the weights in the model are adjusted during pretraining and not just the embedding layers in the case of LM? Word2vec has been a popular approach to learn the input representations and is entirely focused on the embedding layer (not the complete network). With the intuition that pretraining is not exclusive applied on the embedding layers, I thought that pretraining the model on a few input samples would be better than random weight initialization. But I agree with you that my task at hand is probably better suited from a pure supervised learning POV. Apr 17, 2022 at 9:56
• Yes, all the weights of the model are adjusted during pretraining. In the case of W2V it's the 2 linear layers (out of which the first is then used as the "embedding" layer for other tasks) , in the case of Transformer models it's the Embedding layer along with all the other attention layers and MLPs. What model are you thinking to use? Are you trying to learn different representations for your sequence elements? Apr 17, 2022 at 11:32