How do you fine-tune a LLM in theory?

Question

What does it mean to fine-tune a LLM? What can be accomplished with fine-tuning?

I am working on cleaning up messy text (call it tweets) into short clean summaries. How can I take advantage of fine-tuning an LLM like LLaMa 3.1 or any of the other ones out there, to improve the quality of the results?

I am not looking for specific code, but just trying to get a vision of what is theoretically possible.

• How much "training data" do I need? I've seen people throw around "you need 1000's of examples", examples of what? What do I send to the LLM exactly? (In my case, these tweets to be summarized).

I'm wondering if it's worth my time to try and fine-tune a model somehow, given it's just me (solo developer). Or if my time is better spent elsewhere. From my limited knowledge, creating 1000's of examples of tweets and their summaries would take weeks of 8-hour long grinding days, which I just don't have the energy to do. I can code all day, but manually writing the quality summaries to create 1000's of test cases would be energy-draining for sure.

Is that what fine-tuning means? I am new to the idea of fine-tuning. Looking to shed some light on what it is and what I can do with it (and roughly how to fine-tune at a high level).

Answer 1

You won't be able to fine-tune on Groq. But you can upload your own model, which, of course, you can fine-tune beforehand.

Fine-tuning is a process whereby you provide the already trained LLM with additional data and train it further. In order to be able to update the model, at least some layers need to be unfrozen, i.e. their weights are not locked. Typically you would unlock the last couple of layers, keeping the rest of the model weights frozen, before training on the new data. The new model benefits from the general knowledge of the base model, but will hopefully be more attuned to the use case embodied in your new data. The more layers you unlock, the more data you will require to achieve good results.

There are various forms of fine-tuning. With supervised fine-tuning, you provide labelled data. With instruction fine-tuning, your labelled data matches instructions that your LLM should understand. In your case, you would provide a prompt for the action of summarising the tweet, an example tweet and an example clean summary. As @cinch said, you will want synthetic data for this. This can either be from a dataset with clean tweets, which you mess up in a similar fashion to your target dataset; generated by ChatGPT or another LLM; manually curated; or a combination of methods.

You may be able to split the process into cleaning and summarising. In this case you would fine-tune two models, one on each task.

If you want to get an idea of the effort involved, HuggingFace has a tutorial