2
$\begingroup$

I'm playing with an LSTM to generate text. In particular, this one:

https://raw.githubusercontent.com/fchollet/keras/master/examples/lstm_text_generation.py

It works on quite a big demo text set from Nietzsche and says

If you try this script on new data, make sure your corpus has at least ~100k characters. ~1M is better.

This pops up a couple of questions.

A.) If all I want is an AI with a very limited vocabulary where the generate text should be short sentences following a basic pattern.

E.g.

I like blue sky with white clouds

I like yellow fields with some trees

I like big cities with lots of bars

...

Would it then be reasonable to use a much much smaller dataset?

B.) If the dataset really needs to be that big. What if I just repeat the text over and over to reach the recommended minimum? If that would work though, I'd be wondering how that is any different from just taking more iterations of learning with the same shorter text?

Obviously I can play with these two questions myself and in fact I am experimenting with it. One thing I already figured out is that with a shorter text following a basic pattern I can get to a very very low ( ~0.04) quite fast but the predicted text just turns out as gibberish.

My naive explanation for that would be that there are just not enough samples to proof against whether the gibberish actually makes sense or not? But then again I wonder if more iterations or duplicating the content would actually help.

I'm trying to experiment with these questions myself so please don't think I'm just too lazy and are aiming for others to do the work. I'm just looking for more experienced people to give me a better understanding of the mechanics that influence these things.

Improve this question
$\endgroup$
4
  • $\begingroup$ If the patterns are as simple as you describe, you could always try generating a corpus. $\endgroup$
    – NietzscheanAI
    Aug 9, 2016 at 7:04
  • $\begingroup$ You mean, generate a corpus for the learning? Yep, I could generate a corpus where I repeat lots of sentences. I just wonder if that's comparable to just use a smaller dataset (without repeated sentences) and iterate on it more often. $\endgroup$
    – Christoph
    Aug 9, 2016 at 12:50
  • 1
    $\begingroup$ No, I mean: define a grammar template of the form "I like A B with C D" (or similar) and use something like WordNet/ConceptNet to help you generate lots of instances of the template. $\endgroup$
    – NietzscheanAI
    Aug 9, 2016 at 12:53
  • $\begingroup$ Got it. But I'm not so much interested in solving the "lack of data" problem. Rather I'd like to understand why the NN doesn't learn to write (albeit very basic) sentences if it sees a handful of basic sentences over and over. I tried that script with 5 different sentences repeated to a char count of > 1M and still after ~40 iterations it's gibberish. I'm now testing out the original one that's based on a Nietzsche text to see the difference. $\endgroup$
    – Christoph
    Aug 11, 2016 at 7:15

2 Answers 2

Reset to default
2
$\begingroup$

A.) The algorithm can only learn patterns from the data, so if you want it to align to a specific functional form of language, it would make sense to curate your data to only contain sentences of that very structure.

If you tune your hyper-parameters such that the model is not as complex as the default, it should work for datasets that are smaller. But this relies on the observation that the thing you are trying to learn isn't as complex (as opposed to learning from a large corpus of Nietzsche).

B.) What is happening is that you are overfitting the data, such that the LSTM isn't generalizing to your intended goal. In essence, overfitting means that your model is learning irrelevant details that by chance happen to predict the intended goal in the training data. To alleviate this, use a validation set.

And finally (but very importantly), these machine learning techniques don't learn the same way as humans do. To understand why you'd need to learn the math behind NNs, and I can't think of an intuitive explanation as to why.

Improve this answer
$\endgroup$
0
$\begingroup$

LSTMs are good and everything but you can try some exotic methods like Convolutional Neural Network for NLP. Perhaps it would be more suited for this need. Have a look at this.

  1. http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/
  2. http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/
Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .