Soft question here.
I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million parameters and the training data could be from 1GB - 1TB, and that the training could take from about a day to a week. Also, there is an open source GPT here.
My question is, if you consider that ChatGPT is trained on a very large and diverse amount of data, you may think a solo project could never compete with it. However, what if you chose a specialized set of training data that was smaller but a much richer, more reliable knowledge base, like only academic science textbooks, or only English literature, or only Python libraries documentation, and so on?
Could it actually be much more useful because it's open-source, you have freedom of use (unlike ChatGPT's heavy behavioral conditioning from OpenAI), and you can choose what kind of knowledge the transformer contains? If the data is smaller but way, way higher quality, could you just make a library of niche GPTs for any topic you are studying?