Is GPT-3 an early example of strong AI in a narrow setting?

Question

In GPT-2, the large achievement was being able to generate coherent text over a long-form while maintaining context. This was very impressive but for GPT-2 to do new language tasks, it had to be explicitly fine-tuned for the new task.

In GPT-3 (From my understanding), this is no longer the case. It can perform a larger array of language tasks from translation, open domain conversation, summarization, etc., with only a few examples. No explicit fine-tuning is needed.

The actual theory behind GPT-3 is fairly simple, which would not suggest any level of ability other than what would be found in common narrow intelligence systems.

However, looking past the media hype and the news coverage, GPT-3 is not explicitly programmed to "know" how to do these wider arrays of tasks. In fact, with limited examples, it can perform many language tasks quite well and "learn on the fly" so to speak. To me, this does seem to align fairly well with what most people would consider strong AI, but in a narrow context, which is language tasks.

Thoughts? Is GPT-3 an early example of strong AI but in a narrower context?

Answer 1

GPT-3 is based on in-context learning. It’s common wisdom one can hope that bigger models will yield better in-context capabilities. And indeed, this holds true, in the case of GPT-3 175B or "GPT-3".

Neverthless GPT-3 is more powerful than it's predecessors. In some of the tasks, GPT-3 failed miserably. This might be due to the choice to use an autoregressive LM, instead of incorporating bidirectional information (similarly to Bert).

While in-context learning is more straightforward with autoregressive LMs, bidirectional models are known to be better at downstream tasks after fine-tuning.

In the end, training a bidirectional model at the scale of GPT-3 or trying to make bidirectional models work with few-shot learning is a promising direction for future research.

Check out this, this and the paper on Scaling Laws for Neural Language Models.