I am trying to train 2 neural networks - one to compress and the other to decompress real time speech data, and I have a few questions.

First off

Would this even work - trained, of course, on the same type of speech data which would be spoken into the microphone?


Would this be the way in which I pass the data to the NN?
A person is speaking into a microphone, and for every 256 samples of audio data we do the following:

  1. 256 samples of data (input data)
  2. STFT
  3. Abs
  4. MEL spectrogram
  5. Power to dB
  6. Normalization
  7. Neural network inference
  8. Compressed representation (output data)

With the code below, should I:

  1. Use the 256 samples I got from my microphone as y input - nn_input is of shape (n_mels, 3)
  2. Put the 256 samples in a FIFO, which would be the y input - nn_input is of shape (n_mels, 2+FIFO_len)
  3. Something else - I am way off
S = np.abs(librosa.stft(y, hop_length=hop_length, n_fft=hop_length*2))**2
mel_spec = librosa.feature.melspectrogram(S=S, sr=sr, n_mels=n_mels, hop_length=hop_length)
mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max)
nn_input = normalize(mel_spec_db)

NN architecture and layout

I was thinking of using autoencoders for doing this - is there something better, would this be the wrong choice?

Now, here is my biggest issue: how do I lay out the encoder and decoder?

  • Convolutional layer/s followed by smaller dense layers in the encoder, and inverse in the encoder?
    • This would require having history in the input, like in point 2 above? Isn't that pretty stupid with autoencoders, since the output should be the same as the input, and I basically ignore all but 3 elements of the output?
  • Something else
  • $\begingroup$ Please, ask 1 question per post, so posts should not be used to ask multiple questions that you need an answer to solve your big problem/project, but should be used to ask one specific question at a time. Moreover, put your main specific question in the title, so that people immediately understand what your question is. $\endgroup$ – nbro Mar 9 at 11:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.