I’m using Keras LSTM layers and building a model that is trained off ethics text. I have a problem of often over fitting (the network basically remembers my input corpus as it is very small).
I was wondering if anyone has heard of or thought about creating a custom optimizer that takes in the output wordVec but also wordVec’s found next to the same input at a decreasing level of error... bare with me. So instead of the goal to maximise the network towards getting the right next word in a given sequence it is maximising the chance of a correct word across the corpus with the same word.
I feel like this may be an complication but thought it was worth asking.