1
$\begingroup$

To whomever can help out, I appreciate it.

I am currently attempting to detect a signal from background noise. The signal is pretty well known but the background has a lotttt of variability. I've since come to know this problem as Open Set Recognition. Another complicating factor is that the signal mixes with the background noise (think equivalent to a transparent piece of glass in-front of scenery for a picture, or picking out the sound of a pin drop in an office space).

When I started this project, it seemed like the current state of the art in this space was generating Spectrograms and feeding them to a CNN and this is the path I've followed. I'm at a place where I think I've overcome most of the initial problems you might encounter but I'm still not getting good enough results for a project solution.

Here's the overall steps I've gone through:

  1. Generate 17000 ground truth "signals" and 17000 backgrounds (negatives or other classes depending on what nn scheme I'm training)
  2. Generate separate test samples (not training samples but external model validation samples: "blind test") where I take the backgrounds and randomly overlay the signal into it at various intensities.
  3. My first attempt was with a pre-built library training solution (ImageAI) with resnet50 base model. This solution is a multiclass classifier so I had 400 each of the signal + 5 other classes that were the background. It did not work well at classifying the signal. I don't think I ever got this off the ground for two reasons a) My spectrogram pictures were not optimised (waay to large) and b) I couldn't adjust the image input shape via the library. It mostly just ended up classifying one background class.
  4. I then started building my own neural nets. The first reason to make sure my spectrogram input shape was matched in the input shape of the CNN. The second reason was to test various neural net schemes to see what worked best.
  5. The first net I built was a simple feed forward net with a couple of dense layers. This trains to .9998 val_acc. It (like the rest of what I try) produces poor results on my blind tests, in the range of 60% true positive.

    '''
    def build(width, height, depth, classes):
    # initialize the model along with the input shape to be
    # "channels last" and the channels dimension itself
    model = Sequential()
    inputShape = (height, width, depth)
    chanDim = -1
    
    # if we are using "channels first", update the input shape
    # and channels dimension
    if K.image_data_format() == "channels_first":
        inputShape = (depth, height, width)
        chanDim = 1
    model.add(Flatten())
    model.add(Dense(512, input_shape=(inputShape), activation="relu"))
    model.add(Dense(128, activation="relu"))
    model.add(Dense(32, activation="relu"))
    # sigmoid classifier
    model.add(Dense(classes))
    model.add(Activation("sigmoid"))
    
    # return the constructed network architecture
    return model'''
    
  6. I then try a "VGG Light" model. Again, trains to .9999 but gives me only 62% true positive results on my blind tests

    '''
    def build(width, height, depth, classes):
    # initialize the model along with the input shape to be
    # "channels last" and the channels dimension itself
    model = Sequential()
    inputShape = (height, width, depth)
    chanDim = -1
    
    # if we are using "channels first", update the input shape
    # and channels dimension
    if K.image_data_format() == "channels_first":
        inputShape = (depth, height, width)
        chanDim = 1
    
    # CONV => RELU => POOL
    model.add(Conv2D(32, (3, 3), padding="same",
        input_shape=inputShape))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(MaxPooling2D(pool_size=(3, 3)))
    model.add(Dropout(0.25))
    
    # (CONV => RELU) * 2 => POOL
    model.add(Conv2D(64, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(Conv2D(64, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    
    # (CONV => RELU) * 2 => POOL
    model.add(Conv2D(128, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(Conv2D(128, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(GaussianNoise(.05))
    
    # first (and only) set of FC => RELU layers
    model.add(Flatten())
    model.add(Dense(1024))
    model.add(Activation("relu"))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))
    model.add(Dense(512))
    model.add(Activation("relu"))
    model.add(BatchNormalization())
    model.add(Dropout(.5))
    model.add(Dense(128))       
    model.add(Activation("relu"))
    model.add(BatchNormalization())     
    model.add(GaussianDropout(0.5))
    
    # sigmoid classifier
    model.add(Dense(classes))
    model.add(Activation("sigmoid"))
    
    # return the constructed network architecture
    return model'''
    
  7. I then try a "full VGG" net. This again trains to .9999 but only a blind test true positive result of 63%.

    '''
    def build(width, height, depth, classes):
    # initialize the model along with the input shape to be
    # "channels last" and the channels dimension itself
    model = Sequential()
    inputShape = (height, width, depth)
    chanDim = -1
    
    # if we are using "channels first", update the input shape
    # and channels dimension
    if K.image_data_format() == "channels_first":
        inputShape = (depth, height, width)
        chanDim = 1
    
    #CONV => RELU => POOL
    model.add(Conv2D(64, (3, 3), padding="same",
        input_shape=inputShape))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(MaxPooling2D(pool_size=(3, 3)))
    #model.add(Dropout(0.25))
    
    # (CONV => RELU) * 2 => POOL
    model.add(Conv2D(128, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(Conv2D(128, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    #model.add(Dropout(0.25))
    
    # (CONV => RELU) * 2 => POOL
    model.add(Conv2D(256, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(Conv2D(256, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    #model.add(Dropout(0.25))
    
    # (CONV => RELU) * 2 => POOL
    model.add(Conv2D(512, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(Conv2D(512, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    #model.add(Dropout(0.25))
    
    # (CONV => RELU) * 2 => POOL
    model.add(Conv2D(1024, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(Conv2D(1024, (3, 3), padding="same"))
    model.add(Activation("relu"))
    model.add(BatchNormalization(axis=chanDim))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    #model.add(Dropout(0.25))
    model.add(GaussianNoise(.1))
    
    # first (and only) set of FC => RELU layers
    model.add(Flatten())
    model.add(Dense(8192))
    model.add(Activation("relu"))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))
    model.add(Dense(4096))
    model.add(Activation("relu"))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))
    model.add(Dense(1024))
    model.add(Activation("relu"))
    model.add(BatchNormalization()) 
    model.add(GaussianDropout(0.5))
    
    # sigmoid classifier
    model.add(Dense(classes))
    model.add(Activation("sigmoid"))
    
    # return the constructed network architecture
    return model '''
    
  8. All of the above are binary_crossentropy trained in keras.

  9. I've tried multi-class with these models as well but when testing them on the blind test they usually pick the background rather than the signal.

  10. I've also messed around with Autoencoders to try and get the encoder to rebuild the signal well and then compare to known results but haven't been successful yet though I'd be willing to give it another try if everyone thought that might produce better results.

  11. In the beginning I ran into unbalanced classification problems (I was noob) but under all the models shown above the classes all have the same number of samples.

I'm at the point where the larger VGG models trained on 34,000 samples is taking days and I don't see any better results than a basic, feed forward NN that takes 4 minutes to train. Does anyone see the path forward here?

My thanks in advance.

Mecho

p.s. sorry for the formatting flubs, not sure how to correct them.

$\endgroup$
  • 1
    $\begingroup$ have you tried using an adjustable learning rate? Try using ReduceLROnPlateau while monitoring validation loss. documentation is at keras.io/callbacks. Sometimes the validation loss surface is like going down into an increasingly narrower valley. You can get further down into the valley if you reduce the learning rate. I also recommend to use an established model like MobileNet it has about 4 million parameters so it is much faster than VGG and about as accurate. $\endgroup$ – Gerry P Apr 17 at 8:23
  • 1
    $\begingroup$ Also in line 5 you state "This trains to .9998 val_acc." I think you meant training accuracy correct? $\endgroup$ – Gerry P Apr 17 at 8:29
  • 1
    $\begingroup$ Having questions about your problem When you say you have a signal, what kind of signal is it? Is it a waveform like an electrical signal? Is it an image embedded in noisy pixels? If it is an electrical signal embeded in noise use a narrowband bandpass filter. If it is an image with noisy pixels use a noise cancelling auto encoder. $\endgroup$ – Gerry P Apr 17 at 16:40
  • $\begingroup$ @GerryP To answer some of your questions. Yes, .9998 is acc and val_acc with the training data set. My 60% (.6) accuracy is with data the model has never seen before. I have never used adjustable learning rates but I have used very low learning rates (1e-6) with similar results. The problem is an audio signal in background audio "noise" where the background is extremely varied and often at similar or greater amplitude levels to the signal. For using an established model, is this training a custom set with MobileNet as a base? $\endgroup$ – Mecho Engineer Apr 18 at 6:46
0
$\begingroup$

Thanks for the answers. If you are processing an audio signal I think the application of a low pass filter (lpf) would help to enhance the signal to noise ratio. This would help especially if the noise component occupies a large part of the spectrum. If the audio is human speech the majority of the energy is within the 300Hz to 3Khz region. Using a low pass filter with a cutoff frequency of 3Khz would eliminate noise that is in the higher part of the spectrum. You could implement the lpf as a pre-processing function. I am not knowledgeable on the implementation but a search should get you the info you need. I did find an article here. If I recall the process is to convert the time domain signal to the frequency domain using a FFT, then set a cutoff point and reconvert back to the time domain. I also know there are ways to implement that directly in the time domain.Hope this helps. I am also supersized that if you achieve a high validation accuracy that your test set accuracy is so low. Your validation data should be data the network has not seen before just like your test data. Only thing I can think off is that the test data has a very different probability distribution than the training and validation data. How were the various data sets (train, test, validate) selected? Best choice is to select these randomly using something like sklearn train_test_split or Keras ImageDataGenerator flow from directory. Hope this helps.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the additional help. This application is not human speech, unfortunately. I believe that any filtering I could do for the background noise would effectively filter the signal as well. With respect to train, test, and validate, I get train, test from train_test_split. To validate, I take audio clips of the signal and overlay them at various times over background audio (at various intensities) and then run the composited audio clip through the model.predict function. The model has been trained on both the signal and background but just not together. $\endgroup$ – Mecho Engineer Apr 19 at 5:07
0
$\begingroup$

To anyone who reads this, I still haven't solved this completely. At the moment I'm doing a lot better with much cleaner data, using a loss metric that matches what I'm after (F1_Score), using a very deep learning model (a custom Inception Resnet V2 model), use a custom learning rate function that depends on the training round's F1 Score, and every training round computing a F1 Score for various dB of signal/noise test sets with which I compute a model wellness score with which I determine if the model is good enough. Pretty close.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.