What exactly is meant as losing the "competitive moat" when sharing AI models?

Question

I just read this seemingly interesting and important insight from Eliezer Yudkowsky on Twitter, which I copy here for reference:

I don't think people realize what a big deal it is that Stanford retrained a LLaMA model, into an instruction-following form, by cheaply fine-tuning it on inputs and outputs from text-davinci-003.

It means: If you allow any sufficiently wide-ranging access to your AI model, even by paid API, you're giving away your business crown jewels to competitors that can then nearly-clone your model without all the hard work you did to build up your own fine-tuning dataset. If you successfully enforce a restriction against commercializing an imitation trained on your I/O - a legal prospect that's never been tested, at this point - that means the competing checkpoints go up on bittorrent.

I'm not sure I can convey how much this is a brand new idiom of AI as a technology. Let's put it this way:

If you put a lot of work into tweaking the mask of the shoggoth, but then expose your masked shoggoth's API - or possibly just let anyone build up a big-enough database of Qs and As from your shoggoth - then anybody who's brute-forced a core unmasked shoggoth can gesture to your shoggoth and say to their shoggoth "look like that one", and poof you no longer have a competitive moat.

It's like the thing where if you let an unscrupulous potential competitor get a glimpse of your factory floor, they'll suddenly start producing a similar good - except that they just need a glimpse of the inputs and outputs of your factory. Because the kind of good you're producing is a kind of pseudointelligent gloop that gets sculpted; and it costs money and a simple process to produce the gloop, and separately more money and a complicated process to sculpt the gloop; but the raw gloop has enough pseudointelligence that it can stare at other gloop and imitate it.

In other words: The AI companies that make profits will be ones that either have a competitive moat not based on the capabilities of their model, OR those which don't expose the underlying inputs and outputs of their model to customers, OR can successfully sue any competitor that engages in shoggoth mask cloning.

There is more notes in his thread comments.

I don't quite fully understand what he is saying, or the major implications he is talking about. Can you explain in simpler terms or more basic concepts (or metaphors even) what they are saying?

If you allow any sufficiently wide-ranging access to your AI model, even by paid API, you're giving away your business crown jewels to competitors that can then nearly-clone your model without all the hard work you did to build up your own fine-tuning dataset.

So giving API access to your AI model (like ChatGPT is doing? or like LLaMA is doing with its open source model?) is giving away your competitive advantage, I get that. But why/how is it giving away your competitive advantage? How can they clone the model just by using the interface?

competing checkpoints go up on bittorrent

What do they mean by that?

shoggoth

What is this term used for?

and poof you no longer have a competitive moat.

I don't get it by this point. I don't get the remaining examples either.

Can you explain in simpler terms what is being said and why they are suggesting this is important?

Answer 1

Moat in startup lingo means your unique competitive advantage which makes you valuable. What the author is saying is that by listening to enough Q&A you can replicate an entire model, or at least the resemblance of it, therefore removing the moat of these companies. More precisely transforming a GPT like model into a ChatGPT like model, which is optimised for dialogue and is more aligned to a nice behaviour. In cybersecurity lingo, he is saying that an attacker with unlimited access to the API can reverse engineer the function.

Shoggoth is referring to the GPT-like model, often compared to a monster with many faces. Putting a mask on it is the metaphor for doing RLHF on it, that is superficially aligning the model behaviour, while leaving an unexplainable core of the model.

To be clear, I don't think this is very important (as the average tweet), indeed it glosses over the cost of doing this and on the quality of the reverse engineered model. Also it presumes that you have a powerful model to start with (e.g. llama).

In summary he is saying that RLHF can be easily reverse engineered, but without providing clear proofs or quantitative statements.

If you are not familiar with RLHF see How was ChatGPT trained? or How is ChatGPT trained? .