2
$\begingroup$

Soft question here.

I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million parameters and the training data could be from 1GB - 1TB, and that the training could take from about a day to a week. Also, there is an open source GPT here.

My question is, if you consider that ChatGPT is trained on a very large and diverse amount of data, you may think a solo project could never compete with it. However, what if you chose a specialized set of training data that was smaller but a much richer, more reliable knowledge base, like only academic science textbooks, or only English literature, or only Python libraries documentation, and so on?

Could it actually be much more useful because it's open-source, you have freedom of use (unlike ChatGPT's heavy behavioral conditioning from OpenAI), and you can choose what kind of knowledge the transformer contains? If the data is smaller but way, way higher quality, could you just make a library of niche GPTs for any topic you are studying?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Yes, with caveats.

Yes: If the data covers a niche and is very rare, you can indeed fine-tune a large model to your needs.

Caveats: Fine tuning a model still require significant compute. Moreover the largest models to start the fine-tuning with (e.g. GPT-3) are not open-source, even though this may change, so you would settle on a older open-source model. And without the expensive RLHF process to train the agent (paying the human labellers), your agent will be very knowledgeable in your niche area, but overall poorly aligned and potentially toxic.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you very much. I think it would be interesting to have total control over the transformer - oversight or deliberate selection of all data it "knows" - not pre-training on say, the web in general, and then fine-tuning with specialized knowledge. I guess I thought the toxicity of a model was (often) just a reflection of toxic material / debris that fell into its training material, trawled in from the internet at large. I still don't have a great intuition about how much data a transformer needs to perform well... they say quality over quantity. What if trained exclusively on like, OED? $\endgroup$
    – Julius H.
    Jan 30 at 19:06
  • $\begingroup$ If a transformer knew nothing except the entirety of the Oxford English Dictionary, I would hope it must find at least some patterns, and it could be interesting to see at minimum what it is capable of, if it has any interesting use at all. $\endgroup$
    – Julius H.
    Jan 30 at 19:07
  • 1
    $\begingroup$ If you are proposing training from scratch, that would give you the highest quality, but it is going to be expensive (millions). Indeed the network must be large to be good (at least as of today architectures), which means a lot of parameters to train, therefore a lot of compute required. $\endgroup$
    – Rexcirus
    Jan 30 at 20:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .