I find the model very confusing in the input it takes and the output it generates, and some of the layer dimensions didn't seem to match based on what I understood from the wavenet paper, or am I misunderstanding something?
1) What is the input to the WaveNet, isn't this a mel-spectrum input and not just 1 floating point value for raw audio? E.g. the input kernel layer shows as shaped 1x1x128. Isn't the input to the
input_convolution layer the mel-spectrum frames, which are 80 float values * 10,000 max_decoder_steps, so the in_channels for this conv1d layer should be 80 instead of 1?
inference/input_convolution/kernel:0 (float32_ref 1x1x128) [128, bytes: 512]
2) Is there reason for upsampling stride values to be [11, 25], like are the specific numbers 11 and 25 special or relevant in affecting other shapes/dimensions?
inference/ConvTranspose1D_layer_0/kernel:0 (float32_ref 1x11x80x80) [70400, bytes: 281600] inference/ConvTranspose1D_layer_1/kernel:0 (float32_ref 1x25x80x80) [160000, bytes: 640000]
3) Why is the input-channels in residual_block_causal_conv 128 and residual_block_cin_conv 80? What exactly is their inputs? (e.g. is it mel-spectrum or just a raw floating point value?) Is the wavenet-vocoder generating just 1 float value per 1 input mel-spectrum frame of 80 floats?
inference/ResidualConv1DGLU_0/residual_block_causal_conv_ResidualConv1DGLU_0/kernel:0 (float32_ref 3x128x256) [98304, bytes: 393216] inference/ResidualConv1DGLU_0/residual_block_cin_conv_ResidualConv1DGLU_0/kernel:0 (float32_ref 1x80x256) [20480, bytes: 81920]
I was able to print the whole Wavenet network using the print(tf.trainable_variables()), but the model still seems very confusing.