I'm playing with an LSTM to generate text. In particular, this one:
It works on quite a big demo text set from Nietzsche and says
If you try this script on new data, make sure your corpus has at least ~100k characters. ~1M is better.
This pops up a couple of questions.
A.) If all I want is an AI with a very limited vocabulary where the generate text should be short sentences following a basic pattern.
I like blue sky with white clouds
I like yellow fields with some trees
I like big cities with lots of bars
Would it then be reasonable to use a much much smaller dataset?
B.) If the dataset really needs to be that big. What if I just repeat the text over and over to reach the recommended minimum? If that would work though, I'd be wondering how that is any different from just taking more iterations of learning with the same shorter text?
Obviously I can play with these two questions myself and in fact I am experimenting with it. One thing I already figured out is that with a shorter text following a basic pattern I can get to a very very low ( ~0.04) quite fast but the predicted text just turns out as gibberish.
My naive explanation for that would be that there are just not enough samples to proof against whether the gibberish actually makes sense or not? But then again I wonder if more iterations or duplicating the content would actually help.
I'm trying to experiment with these questions myself so please don't think I'm just too lazy and are aiming for others to do the work. I'm just looking for more experienced people to give me a better understanding of the mechanics that influence these things.