I am new to NLP realm. If you have an input text "The price of orange has increased" and output text "Increase the production of orange". Can we make our RNN model to predict the output text? Or what algorithm should I use?
$\begingroup$
$\endgroup$
6
-
$\begingroup$ 2 ways really, both involve converting the input to a 1-hot input. Either convert each character to a 1 hot input (eg, of size 27, 26 in the alphabet, 1 for spaces), or convert each word to a 1 hot vector of the RNN's vocabulary, $\endgroup$– RecessiveSep 9, 2019 at 7:07
-
$\begingroup$ @Recessive Old comment, I know, but: No, those are not the only ways. Another method that is extremely common is to segment text into subwords. Also, you are implying that this tasks needs to be solved with an RNN - which is not the case. $\endgroup$– Mathias MüllerMay 6, 2020 at 7:28
-
$\begingroup$ @MathiasMüller I fail to see where I implied only an RNN could be used, I only responded to the question on how to input the text to an RNN. When you say segment the text into subwords, would those subwords not be encoded in a 1-hot vector? How do you propose inputting ascii characters into a network that deosn't involve 1-hot encoding? $\endgroup$– RecessiveMay 6, 2020 at 17:28
-
$\begingroup$ @Recessive "I fail to see where I implied only an RNN could be used" - you are right, I'm sorry about this! "How do you propose inputting ascii characters into a network that deosn't involve 1-hot encoding?" - my view is as follows: all you have to do is map characters/words/subwords to vocabulary indexes. Then those indexes can be replaced with their word embeddings, without constructing one-hot vectors. $\endgroup$– Mathias MüllerMay 6, 2020 at 18:04
-
$\begingroup$ @MathiasMüller From what I understand, you're saying that the input and output should be an individual number representing the index in the vocabulary? In that case the network would have to use regression to learn, which in my experience has always performed poorly compared to an equivalent solution using classification $\endgroup$– RecessiveMay 7, 2020 at 3:00
|
Show 1 more comment
2 Answers
$\begingroup$
$\endgroup$
In your case you can use either RNN (especially BiLSTM with ELMo and attention mechanism for better accuracy) or Transformer based architectures (the best of them today is BERT). But for both cases you need data to train the model (i.e sequences of input/output like in your question).
I believe the best choice in your case is BERT as it's achieving state of the art performance in most NLP tasks and is already pretrained so you don't need a massive data to retrain the model. Also, BERT is pretrained on "Next Sentence Prediction" and allows "2 separate sentences" as input which helps a lot in your case. The only drawback, compared to other methods, is that you need fine-tuning the model so it's not a completly "ready to use" model.
For more information:
Here is the github repository of BERT: BERT-repo. For quick explanation check: The Illustrated BERT. For detailed explanation see BERT paper: BERT-paper.
$\begingroup$
$\endgroup$
sounds like a job for Sequence-to-sequence or seq2seq
for example tf-seq2seq
you basically use an RNN as an encoder (reduces and sequence of words to a vector), and another RNN as a decoder (takes the encoded vector as in input to generation a new sequence of words).
