I've just recently learned about Text-To-Speech models and how they are trained. Unlike LLMs, to a provided pair (text, speech), a feature vector ${f}$, that was generated by more speech of that speaker and tries to subsume the general voice characteristics, is added. I.e., for a speaker ${S}$, triplets $({text}_i, {speech}_i, {f}_S)$ are used in training (where ${speech}_i$ are speeches from speaker ${S}$). Why? Because speech, compared to text, is much more high-detail-low-content.

I then wondered about training of LLMs. Texts of various sources (scientific, forums, joke books, ...) are used; all trying to fit into one and make one "writer". Hallucinations of LLMs seem to magnify when they are already "on the wrong path", which makes perfect sense: In an article full of nonsense, jokes, wrong data, more of exactly that will follow; so this statistical model of a knowledge base will proceed by giving more of that.

Of course, implicitly, LLMs learn how to write; I may just include in my prompt that "You are a child. Please talk to me as a child." and it will do so. But it was never particularly trained on that.

So the question is: Has there been any work on providing the source, theme, genre, ... of texts provided in training (which should be reasonably possible to add to the training data) as a separate "feature", so as to when prompting that LLM and providing the field of applications, it would generate better results on fewer training data and housr due to less necessary generalization?

(It should be possible to just try this on one's own, e.g., with LoRA, but my expertise and commitment is a bit too low for that, yet.)



You must log in to answer this question.

Browse other questions tagged .