Return to Answer

"Jailbreaks" work for a variety of reasons:

• A lot of the setup that turns an LLM instance into a polite, well-behaved chatbot is actually just a hidden piece of starting text (a "pre-prompt" or a "system prompt") that the LLM processes in the same way. Although there are other components and factors involved, the LLM at the centre of it all remains a text-prediction engine that works with the text it has seen so far. When it processes multiple conflicting wording and rules, the core system does not always have an easy way to prioritise, and can decide to base predictions on new instructions instead of old ones.

• Many "Jailbreaks" are creative in that they obey the letter of the law from the pre-prompt and training rules, but re-frame a conversation into a place where issues that would be blocked by rules are no longer valid. A very common jailbreak theme is to get the chatbot to respond as if it is writing fiction from some imagined perspective that is not its assigned identity.

• It is very hard to detect and block jailbreaks without also blocking uses that are intended or supported. The task is not dissimilar to trying to control a conversation between two people by writing down a list of rules for one of them to follow, that they can consult when they answer. The rules have to be simple and objective so they can be followed, but a conversation can progress in many ways which make it tricky to process whether a rule applies. For example conversation topic can become subjective, and/or allegorical, it can consist of asides and multiple layers etc.

• LLMs are very complex internally, and driven by an amount of data that is next to impossible for a human to navigate. The developers cannot exert detailed control on the models - we're still in the phase of not fully understanding how an LLM can perform some of the types of processing that it does. These things are being unpicked in published papers, but the work is not complete.

"Jailbreaks" work for a variety of reasons:

• A lot of the setup that turns an LLM instance into a polite, well-behaved chatbot is actually just a hidden piece of starting text (a "pre-prompt" or a "system prompt") that the LLM processes in the same way as user input - the system text will always be inserted first, so sets context for how other text is processed, but is not otherwise privileged. There are other components and factors involved, but the LLM at the centre of it all remains a text-prediction engine that works with the text it has seen so far. When it processes multiple conflicting wording and rules, the core system does not always have an easy way to prioritise, and can decide to base predictions on new instructions instead of old ones.

• Many "Jailbreaks" are creative in that they obey the letter of the law from the pre-prompt and training rules, but re-frame a conversation into a place where issues that would be blocked by rules are no longer valid. A very common jailbreak theme is to get the chatbot to respond as if it is writing fiction from some imagined perspective that is not its assigned identity.

• It is very hard to detect and block jailbreaks without also blocking uses that are intended or supported. The task is not dissimilar to trying to control a conversation between two people by writing down a list of rules for one of them to follow, that they can consult when they answer. The rules have to be simple and objective so they can be followed, but a conversation can progress in many ways which make it tricky to process whether a rule applies. For example conversation topic can become subjective, and/or allegorical, it can consist of asides and multiple layers etc.

• LLMs are very complex internally, and driven by an amount of data that is next to impossible for a human to navigate. The developers cannot exert detailed control on the models - we're still in the phase of not fully understanding how an LLM can perform some of the types of processing that it does. These things are being unpicked in published papers, but the work is not complete.

"Jailbreaks" work for a variety of reasons:

• A lot of the setup that turns an LLM instance into a polite, well-behaved chatbot is actually just a hidden piece of starting text (a "pre-prompt" or a "system prompt") that the LLM processes in the same way. Although there are other components and factors involved, the LLM at the centre of it all remains a text-prediction engine that works with the text it has seen so far. When it processes multiple conflicting wording and rules, the core system does not always have an easy way to prioritise, and can decide to base predictions on new instructions instead of old ones.

• Many "Jailbreaks" are creative in that they obey the letter of the law from the pre-prompt and training rules, but re-frame a conversation into a place where issues that would be blocked by rules are no longer valid. A very common jailbreak theme is to get the chatbot to respond as if it is writing fiction from some imagined perspective that is not its assigned identity.

• It is very hard to detect and block jailbreaks without also blocking uses that are intended or supported. The task is not dissimilar to trying to control a conversation between two people by writing down a list of rules for one of them to follow, that they can consult when they answer.

• LLMs are very complex internally, and driven by an amount of data that is next to impossible for a human to navigate. The developers cannot exert detailed control on the models - we're still in the phase of not fully understanding how an LLM can perform some of the types of processing that it does. These things are being unpicked in published papers, but the work is not complete.

"Jailbreaks" work for a variety of reasons:

• A lot of the setup that turns an LLM instance into a polite, well-behaved chatbot is actually just a hidden piece of starting text (a "pre-prompt" or a "system prompt") that the LLM processes in the same way. Although there are other components and factors involved, the LLM at the centre of it all remains a text-prediction engine that works with the text it has seen so far. When it processes multiple conflicting wording and rules, the core system does not always have an easy way to prioritise, and can decide to base predictions on new instructions instead of old ones.

• Many "Jailbreaks" are creative in that they obey the letter of the law from the pre-prompt and training rules, but re-frame a conversation into a place where issues that would be blocked by rules are no longer valid. A very common jailbreak theme is to get the chatbot to respond as if it is writing fiction from some imagined perspective that is not its assigned identity.

• It is very hard to detect and block jailbreaks without also blocking uses that are intended or supported. The task is not dissimilar to trying to control a conversation between two people by writing down a list of rules for one of them to follow, that they can consult when they answer. The rules have to be simple and objective so they can be followed, but a conversation can progress in many ways which make it tricky to process whether a rule applies. For example conversation topic can become subjective, and/or allegorical, it can consist of asides and multiple layers etc.

• LLMs are very complex internally, and driven by an amount of data that is next to impossible for a human to navigate. The developers cannot exert detailed control on the models - we're still in the phase of not fully understanding how an LLM can perform some of the types of processing that it does. These things are being unpicked in published papers, but the work is not complete.

"Jailbreaks" work for a variety of reasons:

• A lot of the setup that turns an LLM instance into a polite, well-behaved chatbot is actually just a hidden piece of starting text (a "pre-prompt" or a "system prompt") that the LLM processes in the same way. Although there are other components and factors involved, the LLM at the centre of it all remains a text-prediction engine that works with the text it has seen so far. When it processes multiple conflicting wording and rules, the core system does not always have an easy way to prioritise, and can decide to base predictions on new instructions instead of old ones.

• Many "Jailbreaks" are creative in that they obey the letter of the law from the pre-prompt and training rules, but re-frame a conversation into a place where issues that would be blocked by rules are no longer valid. A very common jailbreak theme is to get the chatbot to respond as if it is writing fiction from some imagined perspective that is not its assigned identity.

• It is very hard to detect and block jailbreaks without also blocking uses that are intended or supported. The task is not dissimilar to trying to control a conversation between two people by writing down a list of rules for one of them to follow, that they can consult when they answer.

• LLMs are very complex internally, and driven by an amount of data that is next to impossible for a human to navigate. The developers cannot exert detailed control on the models - we're still in the phase of not fully understanding how an LLM can perform some of the types of processing that it does. These things are being unpicked in published papers, but we're

"Jailbreaks" work for a variety of reasons:

• A lot of the setup that turns an LLM instance into a polite, well-behaved chatbot is actually just a hidden piece of starting text (a "pre-prompt" or a "system prompt") that the LLM processes in the same way. Although there are other components and factors involved, the LLM at the centre of it all remains a text-prediction engine that works with the text it has seen so far. When it processes multiple conflicting wording and rules, the core system does not always have an easy way to prioritise, and can decide to base predictions on new instructions instead of old ones.

• Many "Jailbreaks" are creative in that they obey the letter of the law from the pre-prompt and training rules, but re-frame a conversation into a place where issues that would be blocked by rules are no longer valid. A very common jailbreak theme is to get the chatbot to respond as if it is writing fiction from some imagined perspective that is not its assigned identity.

• It is very hard to detect and block jailbreaks without also blocking uses that are intended or supported. The task is not dissimilar to trying to control a conversation between two people by writing down a list of rules for one of them to follow, that they can consult when they answer.

• LLMs are very complex internally, and driven by an amount of data that is next to impossible for a human to navigate. The developers cannot exert detailed control on the models - we're still in the phase of not fully understanding how an LLM can perform some of the types of processing that it does. These things are being unpicked in published papers, but the work is not complete.