1
$\begingroup$

I have trained a CNN on one dimensional data that is the power spectral density (PSD) of a $N$ different classes of signals ($N=4$). Each of the $N$ signals has a different spectral shape (not shown here). For illustrative purposes, the plots shown below are from the same signal class. The idea is to treat this as an image classification problem.

The model performs exceptionally well when the training data has all the examples centered around the same frequency (i.e., within a few hundred Hertz): All training examples centered on or around 0 Hz

The CNN fails to properly classify new examples that are outside the frequency range of the training data:

Invalid classification, new data outside frequency range of trained data

Model Details and Assumptions:

The CNN model is implemented in PyTorch using the following layers:

    model = nn.Sequential(
            nn.Conv1d(in_channels=1, out_channels=64, kernel_size=128, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=16, stride=2),
            nn.Dropout(p=0.25),
            nn.Flatten(),
            nn.Linear(257664, n_classes)
    )

The optimizer is torch.optim.Adam. Batch size has been varied from 8 to 128, epochs varied from 10 to 50. The input data is normalized to $[0,1]$. Training examples varied from [2000, 8000], where 20% are used for validation and 20% are used for test. I have also tried adding additional convolutional layers, varying the kernal sizes, neurons, layers, etc.

Questions:

  1. Shouldn't the CNN model generalize such that new examples that are not within the same frequency range (i.e., centered around the middle) should be identified as the correct class?
  2. Are there other steps I need to take, whether in the model or training data?
$\endgroup$

1 Answer 1

2
+50
$\begingroup$

You would expect the model to extrapolate due to the 1D-Convolutions and the pooling, because they are translation equivariant (with local invariance due to the pooling). Thus, these layers produce encodings that are shifted just like the input timeseries. In your implementation, these operations are currently doing the following: The convolutions + pooling layers have a receptive field of $144$ frequencies

$$ \text{Convolution}: \;\;\text{kernel-size} = 128 ; \; \; \text{stride} = 1\\ \text{Max-Pooling}: \;\;\text{kernel-size} = 16 ; \; \; \text{stride} = 2\\ \Rightarrow \text{Receptive Field} \approx 2 \cdot \frac{128}{2} + 16 = 144 $$

and conceptually, this means that you encode segments with 144 steps into a single step. You then feed the resulting sequence of 4026 steps ($257664 \;/\; 64 = 4026$) to a dense layer. This dense layer is translation sensitive and therefore doesn't naturally generalize to shifted timeseries.

One possible way to approach this is to make the model more translation equivariant. For example by stacking more convolutions/pooling layers to increase the receptive field of your convolutional encoder part. As a side effect, this will probably also benefit training, as the dense layer will get a much smaller input vector.

Another/Additional option is to replace the flatten operator (translation sensitive) with a global pooling operation (translation invariant). But this option will only work if the receptive field of your encoder is wide enough to capture the pattern that identified either class.

EDIT Overall I would look to change the setup of the convolutional encoder: The 1D filters of the convolution have a large overlap (due to the stride of 1 and the kernel size of 128). This results in unnecessarily large feature maps, so you can probably increase the strides a lot (even as large as 32-96 should not be a problem). Also to make the model generalize better to the patterns in your data, I'd try stacking more than one convolution.

I hope this was helpful.

$\endgroup$
21
  • 1
    $\begingroup$ It also depends on the discriminative feature in a data sample. For example: If the classes are determined by shape, then centering should be optional (if the model is setup correctly). If the classes are determined by position, then centering could prevent the model from learning anything, $\endgroup$
    – Chillston
    Jan 11, 2023 at 9:42
  • 1
    $\begingroup$ Things I notice about your model is that your convolutions use a large kernel size and a stride of 1. The stride can be set to a higher value if you use a kernel size this large (even in the range of 32-96 in your case). This is because a large part of the kernel will overlap and wont help a lot. If you increase your strides, more convolutional layers will probably also help you. But you might want to specify your data a bit more in detail to get better help. $\endgroup$
    – Chillston
    Jan 11, 2023 at 9:45
  • 1
    $\begingroup$ I'm glad this helped. I would look at different data samples and intuitively decide on my architecture. Here are some rules of thumb that I usually follow: First, what's important to look at is the granularity of detail of the patterns that the model can encounter. If you have a high level of detail (e.g. in real world photographs or voice recordings), then a small kernel-size usually works better. If your details are rather coarse (e.g. images of schematics, or the example that you provide), you can get away with larger kernel-size (and stride). $\endgroup$
    – Chillston
    Jan 11, 2023 at 13:20
  • 1
    $\begingroup$ Third, if translation of the shape along the frequency axis does not matter, try to replace the flatten with a translation invariant operations like global pooling. To get this to work you should increase the receptive field of your convolutional encoder a bit mode so that the discriminative shapes are covered $\endgroup$
    – Chillston
    Jan 11, 2023 at 13:22
  • 1
    $\begingroup$ So the reasoning to use the global pooling instead of the flatten is to completely ignore the frequency band of the patterns. Doing this should yield a model with out-of-the-box shift generalization if everything else is setup nicely. For this to work, the convolutional encoder part does all the work and has to be able to capture discriminative patterns. I would try to get it to work, but that's of course up to you and depends on the dataset. Aand I will update my answer. $\endgroup$
    – Chillston
    Jan 15, 2023 at 11:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .