1
$\begingroup$

I'm new to AI and machine learning.

To fine-tune GPT-3, I understand that we need a set of training examples that each consist of a single input ("prompt") and its associated output ("completion").

I have prepared a dataset with "prompt" and "completion". And I expect that a fine-tuned model would return the corresponding completion after receiving a prompt in my dataset. But due to some reason, I cannot fine-tune GPT-3 at the moment.

So I plan to fine-tune GPT-Neo (or GPT-J or GPT-NeoX). From this video and this video, it seems that they only accept a dataset containing only "prompt".

Does anyone know how I could modify my dataset with "prompt" and "completion" such that it could be used to fine-tune GPT-Neo?

Improve this question
$\endgroup$
1
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Commented Dec 5, 2022 at 2:24

1 Answer 1

Reset to default
0
$\begingroup$

as far as I can tell training gpt neo with only prompt (if you use the completion as the last part of the prompt) will let you achieve the same results.

You probably notice that, on HuggingFace, there is no way to make a supervised Learning with a text generation model using X as the prompt and y as the completion and the reason is that is not needed. Just add the completion to the prompt and you’ll be fine.

Everything I said is well written and explained inside this HuggingFace notebook: https://github.com/huggingface/notebooks/blob/main/examples/language_modeling.ipynb

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .