I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 characters). So far, I have tried LSTM's with 1-3 layers and inputs as subwords and characters. The results so far are not that great, I am getting ~0.5 crossentropy loss and ~50% accuracy.
My inputs are like a sliding window with prepadding, (eg. (for a batch)
Inputs = [[0,0,0,1], [0,0,1,2]...[n-4,..n-1]],
outputs=[[0,0,1,2], ...[n-3,n-2,n-1,n]]) where 0 is padding,
1 is the start token and
n is the end token. Outputs are 1 hot encoded.
The model is an embedding layer, few lstm and dropout layers, followed by time distributed dense and then a dense layer.
My doubt is, is accuracy a right metric, I am using it because at the end, I am making a classification for 4 output values. Another one is, will a transformer be suitable for this, since I want to generate small sentences, (which are nouns) and models like GPT/ Bert are more suitable for capturing dependency between long sentences.