I’m using Keras LSTM layers and building a model that is trained off ethics text. I have a problem of often over fitting (the network basically remembers my input corpus as it is very small).

I was wondering if anyone has heard of or thought about creating a custom optimizer that takes in the output wordVec but also wordVec’s found next to the same input at a decreasing level of error... bare with me. So instead of the goal to maximise the network towards getting the right next word in a given sequence it is maximising the chance of a correct word across the corpus with the same word.

I feel like this may be an complication but thought it was worth asking.

  • 1
    $\begingroup$ You could implement this in the loss, not through the optimizer $\endgroup$ – mshlis Jun 24 '19 at 22:06
  • $\begingroup$ Happen to have any links? $\endgroup$ – benbyford Jun 25 '19 at 8:59
  • 1
    $\begingroup$ not specifically but just use categorical crossentopy and input a distribution rather than the one hot encoding $\endgroup$ – mshlis Jun 26 '19 at 19:02
  • $\begingroup$ Using a word index in Keras, when you say a distribution what do you mean? $\endgroup$ – benbyford Jul 17 '19 at 8:48
  • $\begingroup$ dont use a word index, because your asking for inputting multiple words rights? so input the vector and there you can define a distribution $\endgroup$ – mshlis Jul 17 '19 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.