0
$\begingroup$

I am new to the field of machine learning, even tho I have solid background in semi-related fields (am control system engineer by trade) and as a hobby project I wanted to work a bit with sound classification, where eventually I would like to have a network that transcribes music - in case of drum it would have to detect when a kick/snare/hi-hat drum has been hit, and for other individual instruments it would need to be able to transcribe them in notes.

At the moment I am trying to solve only the drum classification problem and as far as I understand convolutional networks are the way to go, however one could also solve the problem (probably suboptimally) using RNNs since sound is after all a time sequence and for the purpose of understanding RNNs I would like to follow the approach described in the following article:

https://medium.com/towards-data-science/practical-introduction-to-automation-music-transcription-3ad8ad40eab6

You can see my implementation on the following Link.

I have tried to do the similar thing, albeit on only one of the training sets (hence much less data), however the solution doesnt seem to converge not even close, and I am wondering if it is only about the versatility of the data or am I doing something wrong.

This is not the first time I try to solve a problem using RNN without much success so I am inclined to believe problem may be in my formulation of problem/data.

Same as the author, I have tried to convert each drum sound into a short-term Fourier transform and then generate an array where STFT spectrograms for each time window are vertically stacked together. (Each audio file is converted into a sample_length x 1025 freq_bin array and these are stacked vertically)

Target value is derived from the folder name meaning ['kick', 'snare', 'tom', 'overhead' ...] and then later turned to one-hot representation.

I did not break data down in batches, but as far as I understand that should only affect the speed of fitting the RNN, not the performance itself too much.

Data is hence of (BatchSize = 1, seq_length ~ 4000+ samples, 1025 frequency bins).

Snapshop of the entire batch consisting of one time sequence looks something like what is seen on an image below (I have also inserted periods of silence, labeled as None):

STFT data representation

Somehow the problem doesnt seem to converge into a right solution and typically will always give the same output sequence with minor variations in the softmax output, eventho the spectrum of kick drum is so clearly different from the overhead drum, and early stopping typically occurs:

Epoch 26/200
1/1 [==============================] - 7s 7s/step - loss: 4.9557 -    accuracy: 0.2335 - val_loss: 7.1182 - val_accuracy: 0.1133

I have tried playing with hyper-parameters but not with much luck.

Data used for the training can be found here: https://drive.google.com/drive/folders/1l6lDbER8O_SFRaVGe58781YRK6VprtXm?usp=sharing

$\endgroup$

0

You must log in to answer this question.