How to fine-tune GPT-J with small dataset

Question

I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax

I'm trying to fine-tune GPT-J with a small dataset of ~500 lines:

You are important to me. <|endoftext|>
I love spending time with you. <|endoftext|>
You make me smile. <|endoftext|>
feel so lucky to be your friend. <|endoftext|>
You can always talk to me, even if it’s about something that makes you nervous or scared or sad. <|endoftext|>
etc...

Using the create_finetune_tfrecords.py script (from the repo mentioned above) outputs a file with 2 in it. I understand that means my data has 2 sequences.

I could really use some advice with the .json config file. What hyperparameters do you recommend for this small dataset?

The best I came up with trying to follow the guide:

{
  "layers": 28,
  "d_model": 4096,
  "n_heads": 16,
  "n_vocab": 50400,
  "norm": "layernorm",
  "pe": "rotary",
  "pe_rotary_dims": 64,

  "seq": 2048,
  "cores_per_replica": 8,
  "per_replica_batch": 1,
  "gradient_accumulation_steps": 2,

  "warmup_steps": 1,
  "anneal_steps": 9,
  "lr": 1.2e-4,
  "end_lr": 1.2e-5,
  "weight_decay": 0.1,
  "total_steps": 10,

  "tpu_size": 8,

  "bucket": "chat-app-tpu-bucket-europe",
  "model_dir": "finetune_dir",

  "train_set": "james_bond_1.train.index",
  "val_set": {},

  "eval_harness_tasks": [
  ],

  "val_batches": 2,
  "val_every": 400000,
  "ckpt_every": 1,
  "keep_every": 1,

  "name": "GPT3_6B_pile_rotary",
  "wandb_project": "mesh-transformer-jax",
  "comment": ""
}

The problem is that, when I test the fine-tuned model, I get responses that make no sense:

Answer 1

Far from expert, but at least I can shed some light.

Your dataset is simply too small. Finetuning means you "interfere" with what the GPT-J model sees as important for the continuation of a prompt. Because that is what the model does; continue a prompt in a way that logically makes sense to what it has seen. Since your dataset is very small, especially since the sentences are not even full sentences, things go south.

For one, the create_tf_records.py (that I presume you used) script already filters OUT all senquences that are not long enough (I'm not sure, but this could be why you end up with 2 sequences?). Then the model now has 500 sentences (and maybe even just 2) it sees as the MOST IMPORTANT data to use for its continuation (I think this is where the term "fine-tuning" comes from, not that it's a small influence on the data, it requires "fine" precision 😂). I had to find this out myself the hard way - there is very limited documentation - and I'm still not 100% certain, but quite sure about this part; you trained the model into a brain dead zombie that only slightly remembers all of the knowledge that is stored deeply embedded behind your dataset.

Now, this is a far for complete or really correct explanation, but I hope it gives you some understanding. If you understand more of it or better - please don't forget to let me know :)

What you probably could do to actually get somewhere: Create a script, that based on the sentence you want to "inject", creates more data. So if your happy/positive sentence set is what you want to achieve, then create context around those sentences and spin them. Many times, combined in all kinds of ways. I would even suggest adding in some stuff like the binary contents of an image (look at CLIP, or the v-jax implementation of it by kingoflolz) if you know how to. Some chat generated by another model / chatbot - like GPT-2 or Clara - and just append your sentence at the end. Translate your sentences (deepl is recommended atm I think), mix those in. Pose a question before your sentences; "what would be a good way to cheer someone up or let them know you appreciate them? [insert item from dataset]" and spin those questions. Try to keep repeating yourself to a minimum (but different combinations are fine, small deviations also, but won't teach the model that much more). Paste entire news articles as though it's being quoted in a chat conversation and then put one of your sentences, also will work.

Those are my findings so far. Available for further discussing all of this; I'm also still figuring this thing out.