Currently I'm dealing with an assignment that made us implement the network mentioned in this paper. The network has an architecture similar to this:

Network Architecture

As you can see it uses a Unidirectional RNN (in my case LSTM), which does the many to many sequence prediction task while training, giving LSTM outputs to dense layers with softmax activation. For generating the captions, the network is only given the image at first, and then using the prediction of the image, generates a word, which is then fed to the network along with the generated hidden state, and the model does this recursively to find a unique stop token. Here's the prediction code:

def predict(self, image, max_len = 30):
      output = []
      hidden = None
      inputs = self.encoder(image).unsqueeze(1) # Image features
      for i in range(max_len): # Recursively feed generated words to LSTM 
        lstm_out, hidden = self.decoder.lstm(inputs,hidden)
        output_vocab = self.decoder.fc(lstm_out)    
        output_vocab = F.softmax(output_vocab.squeeze(1), dim=1).detach().cpu().numpy()
        words_indices = output_vocab.argsort(axis=1).squeeze()
        word = words_indices[-1]
        if word == self.unk_token_index:
            word = indices[-2]
        if word == self.end_token_index:
        inputs = self.decoder.embed(torch.LongTensor([[word]]).to(image.device))
      return output

The problem I'm having right now is that I don't know whether this generation scheme works with BiLSTMs. Right now my training loss is way better for the sequence to sequence prediction task than the UniLSTM, but my generated captions are far worse.

This is a sample caption generated by Bi-LSTM:

Sample generated by ‌BiLSTM, picture from Flickr8k dataset.

This is a sample caption generated by UniLSTM:

Sample generated by ‌UniLSTM, picture from Flickr8k dataset.

My training loss for BiLSTM converges to 10e-3, while for UniLSTM it converges to 0.5. But the problem is that even before overfitting, BiLSTM is only generating gibberish.


1 Answer 1


So after doing a bit of research, I finally found out why the model is not working at all when I change the LSTM to Bi-LSTM.

The task of the learning is Next Word Prediction for each cell of LSTM. When you have a Uni-directional LSTM, this is inherently a tough task for the model to learn good representations that can help it generate the next word with enough confident.

What happens when you change the model to a Bi-LSTM is that, if you concatenate the forward and backward values of each cell together, you have now the information of the very next word you wanted to predict via the backward route.

To alleviate this issue, Wang et al. propose to do prediction on forward and backward routes separately while training the data, and for generating, see which route has more confidence in its generated caption.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .