# Changing a CNN-LSTM image captioning architecture to use BiLSTMs

Currently I'm dealing with an assignment that made us implement the network mentioned in this paper. The network has an architecture similar to this:

As you can see it uses a Unidirectional RNN (in my case LSTM), which does the many to many sequence prediction task while training, giving LSTM outputs to dense layers with softmax activation. For generating the captions, the network is only given the image at first, and then using the prediction of the image, generates a word, which is then fed to the network along with the generated hidden state, and the model does this recursively to find a unique stop token. Here's the prediction code:

def predict(self, image, max_len = 30):
output = []
hidden = None
inputs = self.encoder(image).unsqueeze(1) # Image features
for i in range(max_len): # Recursively feed generated words to LSTM
lstm_out, hidden = self.decoder.lstm(inputs,hidden)
output_vocab = self.decoder.fc(lstm_out)
output_vocab = F.softmax(output_vocab.squeeze(1), dim=1).detach().cpu().numpy()
words_indices = output_vocab.argsort(axis=1).squeeze()
word = words_indices[-1]
if word == self.unk_token_index:
word = indices[-2]
output.append(int(word))
if word == self.end_token_index:
break
inputs = self.decoder.embed(torch.LongTensor([[word]]).to(image.device))
return output


The problem I'm having right now is that I don't know whether this generation scheme works with BiLSTMs. Right now my training loss is way better for the sequence to sequence prediction task than the UniLSTM, but my generated captions are far worse.

This is a sample caption generated by Bi-LSTM:

This is a sample caption generated by UniLSTM:

My training loss for BiLSTM converges to 10e-3, while for UniLSTM it converges to 0.5. But the problem is that even before overfitting, BiLSTM is only generating gibberish.