# Heavily mixing signal differentiation from Open Set of backgrounds via CNN

To whomever can help out, I appreciate it.

I am currently attempting to detect a signal from background noise. The signal is pretty well known but the background has a lotttt of variability. I've since come to know this problem as Open Set Recognition. Another complicating factor is that the signal mixes with the background noise (think equivalent to a transparent piece of glass in-front of scenery for a picture, or picking out the sound of a pin drop in an office space).

When I started this project, it seemed like the current state of the art in this space was generating Spectrograms and feeding them to a CNN and this is the path I've followed. I'm at a place where I think I've overcome most of the initial problems you might encounter but I'm still not getting good enough results for a project solution.

Here's the overall steps I've gone through:

1. Generate 17000 ground truth "signals" and 17000 backgrounds (negatives or other classes depending on what nn scheme I'm training)
2. Generate separate test samples (not training samples but external model validation samples: "blind test") where I take the backgrounds and randomly overlay the signal into it at various intensities.
3. My first attempt was with a pre-built library training solution (ImageAI) with resnet50 base model. This solution is a multiclass classifier so I had 400 each of the signal + 5 other classes that were the background. It did not work well at classifying the signal. I don't think I ever got this off the ground for two reasons a) My spectrogram pictures were not optimised (waay to large) and b) I couldn't adjust the image input shape via the library. It mostly just ended up classifying one background class.
4. I then started building my own neural nets. The first reason to make sure my spectrogram input shape was matched in the input shape of the CNN. The second reason was to test various neural net schemes to see what worked best.
5. The first net I built was a simple feed forward net with a couple of dense layers. This trains to .9998 val_acc. It (like the rest of what I try) produces poor results on my blind tests, in the range of 60% true positive.

'''
def build(width, height, depth, classes):
# initialize the model along with the input shape to be
# "channels last" and the channels dimension itself
model = Sequential()
inputShape = (height, width, depth)
chanDim = -1

# if we are using "channels first", update the input shape
# and channels dimension
if K.image_data_format() == "channels_first":
inputShape = (depth, height, width)
chanDim = 1
# sigmoid classifier

# return the constructed network architecture
return model'''

6. I then try a "VGG Light" model. Again, trains to .9999 but gives me only 62% true positive results on my blind tests

'''
def build(width, height, depth, classes):
# initialize the model along with the input shape to be
# "channels last" and the channels dimension itself
model = Sequential()
inputShape = (height, width, depth)
chanDim = -1

# if we are using "channels first", update the input shape
# and channels dimension
if K.image_data_format() == "channels_first":
inputShape = (depth, height, width)
chanDim = 1

# CONV => RELU => POOL
input_shape=inputShape))

# (CONV => RELU) * 2 => POOL

# (CONV => RELU) * 2 => POOL

# first (and only) set of FC => RELU layers

# sigmoid classifier

# return the constructed network architecture
return model'''

7. I then try a "full VGG" net. This again trains to .9999 but only a blind test true positive result of 63%.

'''
def build(width, height, depth, classes):
# initialize the model along with the input shape to be
# "channels last" and the channels dimension itself
model = Sequential()
inputShape = (height, width, depth)
chanDim = -1

# if we are using "channels first", update the input shape
# and channels dimension
if K.image_data_format() == "channels_first":
inputShape = (depth, height, width)
chanDim = 1

#CONV => RELU => POOL
input_shape=inputShape))

# (CONV => RELU) * 2 => POOL

# (CONV => RELU) * 2 => POOL

# (CONV => RELU) * 2 => POOL

# (CONV => RELU) * 2 => POOL

# first (and only) set of FC => RELU layers

# sigmoid classifier

# return the constructed network architecture
return model '''

8. All of the above are binary_crossentropy trained in keras.

9. I've tried multi-class with these models as well but when testing them on the blind test they usually pick the background rather than the signal.

10. I've also messed around with Autoencoders to try and get the encoder to rebuild the signal well and then compare to known results but haven't been successful yet though I'd be willing to give it another try if everyone thought that might produce better results.

11. In the beginning I ran into unbalanced classification problems (I was noob) but under all the models shown above the classes all have the same number of samples.

I'm at the point where the larger VGG models trained on 34,000 samples is taking days and I don't see any better results than a basic, feed forward NN that takes 4 minutes to train. Does anyone see the path forward here?

Mecho

p.s. sorry for the formatting flubs, not sure how to correct them.

• have you tried using an adjustable learning rate? Try using ReduceLROnPlateau while monitoring validation loss. documentation is at keras.io/callbacks. Sometimes the validation loss surface is like going down into an increasingly narrower valley. You can get further down into the valley if you reduce the learning rate. I also recommend to use an established model like MobileNet it has about 4 million parameters so it is much faster than VGG and about as accurate. – Gerry P Apr 17 '20 at 8:23
• Also in line 5 you state "This trains to .9998 val_acc." I think you meant training accuracy correct? – Gerry P Apr 17 '20 at 8:29
• Having questions about your problem When you say you have a signal, what kind of signal is it? Is it a waveform like an electrical signal? Is it an image embedded in noisy pixels? If it is an electrical signal embeded in noise use a narrowband bandpass filter. If it is an image with noisy pixels use a noise cancelling auto encoder. – Gerry P Apr 17 '20 at 16:40
• @GerryP To answer some of your questions. Yes, .9998 is acc and val_acc with the training data set. My 60% (.6) accuracy is with data the model has never seen before. I have never used adjustable learning rates but I have used very low learning rates (1e-6) with similar results. The problem is an audio signal in background audio "noise" where the background is extremely varied and often at similar or greater amplitude levels to the signal. For using an established model, is this training a custom set with MobileNet as a base? – Mecho Engineer Apr 18 '20 at 6:46