I am trying to train an LSTM using CTC loss, but the loss does not decrease when I train it. I have created a minimal example of my issue by creating training data where the network simply has to copy the current input element at each time step. Moreover, I have made the length of the label the same as the length of the input sequence and no adjacent elements in the label sequence the same so that both CTC loss and categorical cross-entropy loss can be used. I found that when using categorical cross-entropy loss the model very quickly converges, whereas when using CTC loss it gets nowhere.

I have uploaded by minimal example to colab. Does anyone know why CTC loss is not working in this case?

  • $\begingroup$ Could you please provide the mathematical formula of the CTC loss that you implemented, to have more context? $\endgroup$ – nbro Jan 16 at 21:34
  • 1
    $\begingroup$ I did not implement the loss function myself, but rather am using a built-in function from tensorflow, tf.nn.ctc_loss. The mathematical formulation of this is available in the paper Connectionist Temporal Classification - Labeling Unsegmented Sequence Data with Recurrent Neural Networks. $\endgroup$ – Cameron Martin Jan 17 at 14:56
  • $\begingroup$ Thanks! I would like to note that programming issues are off-topic here. So, next time, if you think your problem is a bug in your code and not a conceptual issue, then you should ask your question on Stack Overflow. Here, we focus on the theoretical, philosophical, and social aspects of Artificial Intelligence. Take a look at ai.stackexchange.com/help/on-topic. $\endgroup$ – nbro Jan 17 at 17:18
  • $\begingroup$ Noted. Thanks for the direction. $\endgroup$ – Cameron Martin Jan 18 at 2:01

The problem was the dimensions of the logit_length argument to tf.nn.ctc_loss was incorrect.

It was this:

tf.repeat(tf.shape(y_pred)[-1], tf.shape(y_pred)[0])

But it should have been

tf.repeat(tf.shape(y_pred)[-2], tf.shape(y_pred)[0])

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.