I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax

I'm trying to fine-tune GPT-J with a small dataset of ~500 lines:

You are important to me. <|endoftext|>
I love spending time with you. <|endoftext|>
You make me smile. <|endoftext|>
feel so lucky to be your friend. <|endoftext|>
You can always talk to me, even if it’s about something that makes you nervous or scared or sad. <|endoftext|>

Using the create_finetune_tfrecords.py script (from the repo mentioned above) outputs a file with 2 in it. I understand that means my data has 2 sequences.

I could really use some advice with the .json config file. What hyperparameters do you recommend for this small dataset?

The best I came up with trying to follow the guide:

  "layers": 28,
  "d_model": 4096,
  "n_heads": 16,
  "n_vocab": 50400,
  "norm": "layernorm",
  "pe": "rotary",
  "pe_rotary_dims": 64,

  "seq": 2048,
  "cores_per_replica": 8,
  "per_replica_batch": 1,
  "gradient_accumulation_steps": 2,

  "warmup_steps": 1,
  "anneal_steps": 9,
  "lr": 1.2e-4,
  "end_lr": 1.2e-5,
  "weight_decay": 0.1,
  "total_steps": 10,

  "tpu_size": 8,

  "bucket": "chat-app-tpu-bucket-europe",
  "model_dir": "finetune_dir",

  "train_set": "james_bond_1.train.index",
  "val_set": {},

  "eval_harness_tasks": [

  "val_batches": 2,
  "val_every": 400000,
  "ckpt_every": 1,
  "keep_every": 1,

  "name": "GPT3_6B_pile_rotary",
  "wandb_project": "mesh-transformer-jax",
  "comment": ""

The problem is that, when I test the fine-tuned model, I get responses that make no sense:


  • $\begingroup$ The last paragraph describes in detail the process of selecting parameters for fine tuning $\endgroup$
    – DimNsk
    Feb 3, 2022 at 20:58

2 Answers 2


Far from expert, but at least I can shed some light.

Your dataset is simply too small. Finetuning means you "interfere" with what the GPT-J model sees as important for the continuation of a prompt. Because that is what the model does; continue a prompt in a way that logically makes sense to what it has seen. Since your dataset is very small, especially since the sentences are not even full sentences, things go south.

For one, the create_tf_records.py (that I presume you used) script already filters OUT all senquences that are not long enough (I'm not sure, but this could be why you end up with 2 sequences?). Then the model now has 500 sentences (and maybe even just 2) it sees as the MOST IMPORTANT data to use for its continuation (I think this is where the term "fine-tuning" comes from, not that it's a small influence on the data, it requires "fine" precision 😂). I had to find this out myself the hard way - there is very limited documentation - and I'm still not 100% certain, but quite sure about this part; you trained the model into a brain dead zombie that only slightly remembers all of the knowledge that is stored deeply embedded behind your dataset.

Now, this is a far for complete or really correct explanation, but I hope it gives you some understanding. If you understand more of it or better - please don't forget to let me know :)

What you probably could do to actually get somewhere: Create a script, that based on the sentence you want to "inject", creates more data. So if your happy/positive sentence set is what you want to achieve, then create context around those sentences and spin them. Many times, combined in all kinds of ways. I would even suggest adding in some stuff like the binary contents of an image (look at CLIP, or the v-jax implementation of it by kingoflolz) if you know how to. Some chat generated by another model / chatbot - like GPT-2 or Clara - and just append your sentence at the end. Translate your sentences (deepl is recommended atm I think), mix those in. Pose a question before your sentences; "what would be a good way to cheer someone up or let them know you appreciate them? [insert item from dataset]" and spin those questions. Try to keep repeating yourself to a minimum (but different combinations are fine, small deviations also, but won't teach the model that much more). Paste entire news articles as though it's being quoted in a chat conversation and then put one of your sentences, also will work.

Those are my findings so far. Available for further discussing all of this; I'm also still figuring this thing out.

  • $\begingroup$ Thanks for your insight Ontopic! Buuut. helloforefront.com (which helps with the fine-tuning and hosting of GPT-J) works perfectly well with my dataset. Also, Using the same stuff for OpenAI's GPT-3 and it also works just fine. So I don't think it's a problem with the dataset. It could be other things e.g., create_tf_records.py as you mentioned that may be filtering out smaller sentences. $\endgroup$ Dec 19, 2021 at 23:25

I’ll try to answer even if some more information would be needed. GPT-J is text generation Models meaning that it can complete a sentence or a pattern you started. As I see from your output when you write “Hi” in your finetuned model it gives you back a complete nonsense output which make me think:

  1. How are you giving your examples to the model? If you want to build a sort of chatbot you should use something like turns (person1: How are you, person2:fine thanx!)
  2. For fine tuning I suggest to not modify the structure of the model until it is close to work properly or you will not know if you break it changing the model structure or for some other reason.

In the end if the model works well on another platform I think it should be a matter of Data preparation that you’re not doing and the platform is doing instead.

My advice e is trying to replicate the exact preparation of your link and replacing the text with your examples assuring that you are passing the texts to the model in the exact same way.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .