Appropriate metric and approach for natural language generation for small sentences

I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 characters). So far, I have tried LSTM's with 1-3 layers and inputs as subwords and characters. The results so far are not that great, I am getting ~0.5 crossentropy loss and ~50% accuracy.

My inputs are like a sliding window with prepadding, (eg. (for a batch) Inputs = [[0,0,0,1], [0,0,1,2]...[n-4,..n-1]], outputs=[[0,0,1,2], ...[n-3,n-2,n-1,n]]) where 0 is padding, 1 is the start token and n is the end token. Outputs are 1 hot encoded.

The model is an embedding layer, few lstm and dropout layers, followed by time distributed dense and then a dense layer.

My doubt is, is accuracy a right metric, I am using it because at the end, I am making a classification for 4 output values. Another one is, will a transformer be suitable for this, since I want to generate small sentences, (which are nouns) and models like GPT/ Bert are more suitable for capturing dependency between long sentences.

• Hi. Can please put your main question in the title? – nbro Sep 13 '20 at 15:25
• @nbro updated, is that okay? – ctlr Sep 13 '20 at 15:28
• That seems more explicit and specific. Thanks! – nbro Sep 13 '20 at 15:28