0
$\begingroup$

Given a corpus of data like a log of Slack conversations, I want to be able to use this and generate fictitious conversations e.g., given 10 conversations, I want to be able to scale this up to 100 conversations. Is there a toolkit to do this? I tried searching but could not find anything. I understand the most naive approach would be to use a pre-trained LLM, give the corpus to it as input and ask it to do the job. I haven't tried this but the real corpus will be large enough such that it won't fit in context window of a pre-trained LLM.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

you mean the whole corpus or just a chat inside a corpus? a simple chat on slack from the start to end usually shouldn't be too long. of your whole data could be large.

To (re-)train your model, you need to split your data into some chunks. length of each chunks could be e.g 4k to 16k token depending on the (maximum) context size of your pretrained LLM.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .