Currently I'm dealing with an assignment that made us implement the network mentioned in this paper. The network has an architecture similar to this:
As you can see it uses a Unidirectional RNN (in my case LSTM), which does the many to many sequence prediction task while training, giving LSTM outputs to dense layers with softmax activation. For generating the captions, the network is only given the image at first, and then using the prediction of the image, generates a word, which is then fed to the network along with the generated hidden state, and the model does this recursively to find a unique stop token. Here's the prediction code:
def predict(self, image, max_len = 30): output =  hidden = None inputs = self.encoder(image).unsqueeze(1) # Image features for i in range(max_len): # Recursively feed generated words to LSTM lstm_out, hidden = self.decoder.lstm(inputs,hidden) output_vocab = self.decoder.fc(lstm_out) output_vocab = F.softmax(output_vocab.squeeze(1), dim=1).detach().cpu().numpy() words_indices = output_vocab.argsort(axis=1).squeeze() word = words_indices[-1] if word == self.unk_token_index: word = indices[-2] output.append(int(word)) if word == self.end_token_index: break inputs = self.decoder.embed(torch.LongTensor([[word]]).to(image.device)) return output
The problem I'm having right now is that I don't know whether this generation scheme works with BiLSTMs. Right now my training loss is way better for the sequence to sequence prediction task than the UniLSTM, but my generated captions are far worse.
This is a sample caption generated by Bi-LSTM:
This is a sample caption generated by UniLSTM:
My training loss for BiLSTM converges to 10e-3, while for UniLSTM it converges to 0.5. But the problem is that even before overfitting, BiLSTM is only generating gibberish.