Many-shot jailbreaking is a jailbreaking technique in large language models(LLMs). Jailbreaking technique is the way to by pass the safety rules that set by LLMs developers.
Many-shot jailbreaking creates a fake dialogue between the human and AI assistant and give the harmful query as the last query in the same prompt.
In-context learning helps for many-shot jailbreaking. In-context learning means LLMs learns only using the information that given in the prompt. Larger language models are good at in-context learning because they only need a shorter prompt which leads to larger language models can produce harmful response in many-shot jailbreaking for shorter prompt. And also long-context window leads to many-shot attacks. Latest released LLMs has context window which has 1M token size.
Solutions for many-shot jailbreaking
- Classify and modify the prompt before it send to the language model. (The similar method is used to identify election related queries.)
- The length of the context window should be reduced to overcome the many-shot jailbreaking attacks.
- LLMs are finetuned to reject to responds prompts that appear as many-shot jailbreaking attacks.