I'm trying to train a model that would generate stories. I have a dataset of 2000 stories prepared. They are tokenized and one-hot encoded. I can't load them all at once as a one big dataset, because of memory limits.

What would be the best way to fit my network so that i can reset the states after each story?

I tried doing it in a nested for loop (for epoch/for story: model.fit) but it's working really slow cause it takes 3 seconds to fit a single story but almost 10 to load the next file and setup model.fit again.

  • $\begingroup$ What is the largest size, that your model can accept? I would suggest to form a batch of stories as a tensor, and the feed it to the network. Like 100, 500 stories, whatever fits into your memory $\endgroup$ Aug 28 '21 at 19:14
  • $\begingroup$ Would probably have to try different sizes and guess what would fit since the stories have different lenghts. The issue is that I want to reset my network after each story so that it models the structure. But if I made a batch of for example 10 stories, then as far as i know i can only reset per batch so I wouldn't be able to reset after each of the stories. Im wondering if there could be a way to create for example a data generator that would feed stories (right now a story is 10-20 batches ) with resetting after each. $\endgroup$
    – ymtyroni
    Aug 29 '21 at 11:34
  • $\begingroup$ I'm guessing that the best way would be to feed a story in one batch but I don't think my memory can handle that since my vocab size is 10k $\endgroup$
    – ymtyroni
    Aug 29 '21 at 11:36
  • $\begingroup$ Nevermind, I realised I'm able to create a data generator that will load a whole story as one batch and it fits in my memory. Don't know why I haven't thought about it before. Thanks $\endgroup$
    – ymtyroni
    Aug 29 '21 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.