Heavily mixing signal differentiation from Open Set of backgrounds via CNN

Question

I am currently attempting to detect a signal from background noise. The signal is pretty well known but the background has a lot of variability. I've since come to know this problem as Open Set Recognition. Another complicating factor is that the signal mixes with the background noise (think equivalent to a transparent piece of glass in-front of scenery for a picture, or picking out the sound of a pin drop in an office space).

When I started this project, it seemed like the current state of the art in this space was generating Spectrograms and feeding them to a CNN and this is the path I've followed. I'm at a place where I think I've overcome most of the initial problems you might encounter but I'm still not getting good enough results for a project solution.

Here's the overall steps I've gone through:

0. Generate 17000 ground truth "signals" and 17000 backgrounds (negatives or other classes depending on what nn scheme I'm training)

1. Generate separate test samples (not training samples but external model validation samples: "blind test") where I take the backgrounds and randomly overlay the signal into it at various intensities.

2. My first attempt was with a pre-built library training solution (ImageAI) with resnet50 base model. This solution is a multiclass classifier so I had 400 each of the signal + 5 other classes that were the background. It did not work well at classifying the signal. I don't think I ever got this off the ground for two reasons a) My spectrogram pictures were not optimised (waay to large) and b) I couldn't adjust the image input shape via the library. It mostly just ended up classifying one background class.

3. I then started building my own neural nets. The first reason to make sure my spectrogram input shape was matched in the input shape of the CNN. The second reason was to test various neural net schemes to see what worked best.

4. The first net I built was a simple feed forward net with a couple of dense layers. This trains to .9998 val_acc. It (like the rest of what I try) produces poor results on my blind tests, in the range of 60% true positive.

   def build(width, height, depth, classes):
        # initialize the model along with the input shape to be
        # "channels last" and the channels dimension itself
        model = Sequential()
        inputShape = (height, width, depth)
        chanDim = -1
   
        # if we are using "channels first", update the input shape
        # and channels dimension
        if K.image_data_format() == "channels_first":
            inputShape = (depth, height, width)
            chanDim = 1
        model.add(Flatten())
        model.add(Dense(512, input_shape=(inputShape),activation="relu"))
        model.add(Dense(128, activation="relu"))
        model.add(Dense(32, activation="relu"))
        # sigmoid classifier
        model.add(Dense(classes))
        model.add(Activation("sigmoid"))
   
        # return the constructed network architecture
        return model  
   

5. I then try a "VGG Light" model. Again, trains to .9999 but gives me only 62% true positive results on my blind tests

    def build(width, height, depth, classes):
        # initialize the model along with the input shape to be
        # "channels last" and the channels dimension itself
        model = Sequential()
        inputShape = (height, width, depth)
        chanDim = -1
   
        # if we are using "channels first", update the input shape
        # and channels dimension
        if K.image_data_format() == "channels_first":
            inputShape = (depth, height, width)
            chanDim = 1
   
        # CONV => RELU => POOL
        model.add(Conv2D(32, (3, 3), padding="same",            input_shape=inputShape))
        model.add(Activation("relu"))
        model.add(BatchNormalization(axis=chanDim))
        model.add(MaxPooling2D(pool_size=(3, 3)))
        model.add(Dropout(0.25))
   
        # (CONV => RELU) * 2 => POOL
        model.add(Conv2D(64, (3, 3), padding="same"))
        model.add(Activation("relu"))
        model.add(BatchNormalization(axis=chanDim))
        model.add(Conv2D(64, (3, 3), padding="same"))
        model.add(Activation("relu"))
        model.add(BatchNormalization(axis=chanDim))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.25))
   
        # (CONV => RELU) * 2 => POOL
        model.add(Conv2D(128, (3, 3), padding="same"))
        model.add(Activation("relu"))
        model.add(BatchNormalization(axis=chanDim))
        model.add(Conv2D(128, (3, 3), padding="same"))
        model.add(Activation("relu"))
        model.add(BatchNormalization(axis=chanDim))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.25))
        model.add(GaussianNoise(.05))
   
        # first (and only) set of FC => RELU layers
        model.add(Flatten())
        model.add(Dense(1024))
        model.add(Activation("relu"))
        model.add(BatchNormalization())
        model.add(Dropout(0.5))
        model.add(Dense(512))
        model.add(Activation("relu"))
        model.add(BatchNormalization())
        model.add(Dropout(.5))
        model.add(Dense(128))       
        model.add(Activation("relu"))
        model.add(BatchNormalization())     
        model.add(GaussianDropout(0.5))
   
        # sigmoid classifier
        model.add(Dense(classes))
        model.add(Activation("sigmoid"))
   
        # return the constructed network architecture
        return model
   

6. I then try a "full VGG" net. This again trains to .9999 but only a blind test true positive result of 63%.

   def build(width, height, depth, classes):
       # initialize the model along with the input shape to be
       # "channels last" and the channels dimension itself
       model = Sequential()
       inputShape = (height, width, depth)
       chanDim = -1
   
       # if we are using "channels first", update the input shape
       # and channels dimension
       if K.image_data_format() == "channels_first":
           inputShape = (depth, height, width)
           chanDim = 1
   
       #CONV => RELU => POOL
       model.add(Conv2D(64, (3, 3), padding="same", input_shape=inputShape))
       model.add(Activation("relu"))
       model.add(BatchNormalization(axis=chanDim))
       model.add(MaxPooling2D(pool_size=(3, 3)))
       #model.add(Dropout(0.25))
   
       # (CONV => RELU) * 2 => POOL
       model.add(Conv2D(128, (3, 3), padding="same"))
       model.add(Activation("relu"))
       model.add(BatchNormalization(axis=chanDim))
       model.add(Conv2D(128, (3, 3), padding="same"))
       model.add(Activation("relu"))
       model.add(BatchNormalization(axis=chanDim))
       model.add(MaxPooling2D(pool_size=(2, 2)))
       #model.add(Dropout(0.25))
   
       # (CONV => RELU) * 2 => POOL
       model.add(Conv2D(256, (3, 3), padding="same"))
       model.add(Activation("relu"))
       model.add(BatchNormalization(axis=chanDim))
       model.add(Conv2D(256, (3, 3), padding="same"))
       model.add(Activation("relu"))
       model.add(BatchNormalization(axis=chanDim))
       model.add(MaxPooling2D(pool_size=(2, 2)))
       #model.add(Dropout(0.25))
   
       # (CONV => RELU) * 2 => POOL
       model.add(Conv2D(512, (3, 3), padding="same"))
       model.add(Activation("relu"))
       model.add(BatchNormalization(axis=chanDim))
       model.add(Conv2D(512, (3, 3), padding="same"))
       model.add(Activation("relu"))
       model.add(BatchNormalization(axis=chanDim))
       model.add(MaxPooling2D(pool_size=(2, 2)))
       #model.add(Dropout(0.25))
   
       # (CONV => RELU) * 2 => POOL
       model.add(Conv2D(1024, (3, 3), padding="same"))
       model.add(Activation("relu"))
       model.add(BatchNormalization(axis=chanDim))
       model.add(Conv2D(1024, (3, 3), padding="same"))
       model.add(Activation("relu"))
       model.add(BatchNormalization(axis=chanDim))
       model.add(MaxPooling2D(pool_size=(2, 2)))
       #model.add(Dropout(0.25))
       model.add(GaussianNoise(.1))
   
       # first (and only) set of FC => RELU layers
       model.add(Flatten())
       model.add(Dense(8192))
       model.add(Activation("relu"))
       model.add(BatchNormalization())
       model.add(Dropout(0.5))
       model.add(Dense(4096))
       model.add(Activation("relu"))
       model.add(BatchNormalization())
       model.add(Dropout(0.5))
       model.add(Dense(1024))
       model.add(Activation("relu"))
       model.add(BatchNormalization()) 
       model.add(GaussianDropout(0.5))
   
       # sigmoid classifier
       model.add(Dense(classes))
       model.add(Activation("sigmoid"))
   
       # return the constructed network architecture
       return model
   

7. All of the above are binary_crossentropy trained in keras.

8. I've tried multi-class with these models as well but when testing them on the blind test they usually pick the background rather than the signal.

9. I've also messed around with Autoencoders to try and get the encoder to rebuild the signal well and then compare to known results but haven't been successful yet though I'd be willing to give it another try if everyone thought that might produce better results.

10. In the beginning I ran into unbalanced classification problems (I was noob) but under all the models shown above the classes all have the same number of samples.

I'm at the point where the larger VGG models trained on 34,000 samples is taking days and I don't see any better results than a basic, feed forward NN that takes 4 minutes to train.

Does anyone see the path forward here?

Answer 1

Thanks for the answers. If you are processing an audio signal I think the application of a low pass filter (lpf) would help to enhance the signal to noise ratio. This would help especially if the noise component occupies a large part of the spectrum. If the audio is human speech the majority of the energy is within the 300Hz to 3Khz region. Using a low pass filter with a cutoff frequency of 3Khz would eliminate noise that is in the higher part of the spectrum. You could implement the lpf as a pre-processing function. I am not knowledgeable on the implementation but a search should get you the info you need. I did find an article here. If I recall the process is to convert the time domain signal to the frequency domain using a FFT, then set a cutoff point and reconvert back to the time domain. I also know there are ways to implement that directly in the time domain.Hope this helps. I am also supersized that if you achieve a high validation accuracy that your test set accuracy is so low. Your validation data should be data the network has not seen before just like your test data. Only thing I can think off is that the test data has a very different probability distribution than the training and validation data. How were the various data sets (train, test, validate) selected? Best choice is to select these randomly using something like sklearn train_test_split or Keras ImageDataGenerator flow from directory. Hope this helps.

Answer 2

To anyone who reads this, I still haven't solved this completely. At the moment I'm doing a lot better with much cleaner data, using a loss metric that matches what I'm after (F1_Score), using a very deep learning model (a custom Inception Resnet V2 model), use a custom learning rate function that depends on the training round's F1 Score, and every training round computing a F1 Score for various dB of signal/noise test sets with which I compute a model wellness score with which I determine if the model is good enough. Pretty close.