can I add to a language model a prompt with output example?

Question

I want to finetune GPT2 to extract relevant data from a given text. So for (a trivial) example, given the text "the car was manufactured in X, can reach Y km/h, and has Z horse powers", my desired output would be manufacturer: X, max speed:Y, horsepowers:Z.

I don't have a lot of labeled data, so I thought it would be reasonable to take every training sample and add to it a prefix than contains an actual example. That is - Instead of providing the model with the text

INPUT: a well-known brand, the new X, can reach Y km/h, and has Z horsepower

OUTPUT:

and expect the model to understand how to fill in the details, I would provide a longer prompt that contains an actual example like

INPUT: the new BMW can reach up to 200 kmh. Even though the previous model disappointed the users, the brand-new one rocks a thousand engine horsepower

OUTPUT: Manufacturer: BMW, Max Speed: 200, Horse Powers: thousand

INPUT: though is mostly considered an outdated version of the cls200, the new Mercedes has the capabilities of reaching up to 100kmh in turns and 300kmh overall

OUTPUT:

Is this considered a common way of engineering the prompt? Do notice that the first provided example is the same for every training sample.

Answer 1

Yes, it is possible to finetune GPT2 to extract relevant data from a given text. For your example, you could add a prompt that looks for a manufacturer, max speed, and horsepower, and then provide an output example that includes that information.

It is possible to add a prefix to every training sample that contains an actual example. This could help the model learn from data that has fewer labels.

INPUT: though is mostly considered an outdated version of the cls 200, the new Mercedes has the capabilities of reaching up to 100kmh in turns and 300kmh overall? The OUTPUT would be the capabilities of the new Mercedes, including its max speed and ability to handle turns.

Yes, this is considered a common way of engineering the prompt.

Providing the same example for every training sample also help the model learn from data that has fewer labels.