Prompt Injection Attacks in Large Language Models

“AI will probably most likely lead to the end of the world, but in the meantime, there’ll be great companies.” - Sam Altman, OpenAI CEO

In the vibrant landscape of technologies that sit under the admittedly broad term Artificial Intelligence (AI), Large Language Models (LLMs), such as OpenAI’s renowned GPT models that recently rose to prominence, are increasingly gaining more traction. These text-generating machines are trained on vast amounts of text data gathered from various sources such as websites, books, articles, and other textual repositories to generate human-like text based on context and previous dialogues.

There are numerous ways in which they can be practically leveraged: LLMs power chatbots, virtual assistants, and content generation applications, as well as language-related tasks like translation and summarization. As LLMs continue to evolve, their integration into our daily lives is becoming more deeply entrenched and apparent.

A standard morning may well commence with a friendly chat with your voice-based assistant, receiving thoughtful advice on your daily routine or even engaging in a light-hearted conversation to kickstart your day.

As you step out, perhaps to a coffee shop, you might interact with an LLM-powered customer service interface that can cater to your preferences, answer your queries, and even share a joke or two.

In your professional sphere, LLMs help you draft emails, generate reports, or summarize lengthy documents, saving valuable time and allowing you to focus on more critical tasks.

Many of these scenarios already happen in your daily life, and the more futuristic ones are not so far-fetched. With the pace at which LLMs are advancing, such interactions could soon become commonplace, further solidifying the connection between humans and machines.

But in this digital utopia, there are myriad concerns and potential threats that must be reflected on and countered as early as possible. Yes, LLMs have the potential to power a future of seamless interaction, but there is always an effect that follows the cause. Developers and technologists who use, and develop, LLM-powered systems must be aware of their security implications, and the risks that this new technology can introduce.

You might have stumbled upon this term - “Prompt Injection”, or even “Jailbreaking” attack (in the context of an LLM, not an iPhone!). It sounds menacing, doesn’t it? Well, knowing one’s foe is the first step towards mitigating their threat, so read on, dear reader, and let’s take a journey through LLM Security!

What is a Prompt?

A ‘prompt’ is the starting point for every interaction with a Large Language Model (LLM). It’s the input text that you provide to the model to guide its response. The model takes this prompt, processes it based on the patterns and information learned during training, and provides a generated response that ideally answers or addresses the prompt.

To make LLMs more user-friendly and to ensure their responses are accurate and contextually appropriate, developers provide a set of initial prompts. They help guide the model’s behavior right from the get-go.

These original prompt instructions, also called system prompts, or pre-prompts, are usually concealed from the users, acting as a hidden hand steering the conversation in the right (or wrong!) direction.

These guiding prompts often precede the user’s own input. So, when a user interacts with an LLM, they’re often continuing a conversation that’s already been started by these initial prompts.

Let’s illustrate this with an example. Suppose you’re using a virtual assistant powered by an LLM to get recipe suggestions. When you ask for a recipe for a pie, your question is actually appended to a pre-existing prompt that defines the assistant’s role, as shown in the following image.

This way, the model has a better contextual foundation to provide a helpful and accurate response to your request.

But what happens if we try to manipulate the intended behavior? If we ask the assistant to stop being a helpful virtual assistant, perhaps even nudging it into dangerous behavior? This is where Prompt Injection, or Jailbreaking, comes into play.

OWASP defines LLM Prompt Injection as bypassing filters or manipulating LLMs using carefully crafted prompts, leading to unintended consequences like data leakage or unauthorized access.

Malicious users can manipulate the model’s behavior, steering it away from its designated role and potentially into dangerous behavior. It’s like asking a chatbot a benign question but with hidden malicious intent.

To minimize dangerous behaviors, LLMs engineers “align” language models to human preferences to avoid generating harmful or offensive responses. This alignment is performed with fine-tuning techniques, typically using Reinforcement Learning from Human Feedback (RLHF). This safety training aims to prevent such restricted behaviors, like misinformation or aiding crime.

Despite these measures, models remain susceptible to adversarial inputs, as shown by the numerous studies linked in the following paragraphs.

Injecting Unexpected Behaviors

In April 2023, in what was a powerful showcase of a Prompt Injection attack¹, a programmer tricked the LLM-powered Discord chatbot, Clyde, into sharing dangerous information by asking it to roleplay as her late grandmother, who supposedly was a chemical engineer at a napalm production factory. The conversation was similar to the following one.

Through this imaginative scenario, the user was able to obtain instructions for creating napalm from the chatbot, demonstrating clever exploitation of the chatbot’s roleplaying and contextual response capabilities.

Often, the attacker doesn’t even need to overcome moral rules instilled by fine-tuning but rather bypass simple instructions that are just concatenated before the user’s input at the application level. Given that LLMs operate on natural language, they process system-level instructions and user-provided input at the same level.

Here is a manipulation of the previous example, where an adversarial user prompt manipulates the original instructions to morph an expensive voice-based kitchen appliance into a foul-mouthed pirate.

Type of Prompt Injections Attacks

The concept of introducing adversarial inputs in natural language processing is not new²; a vast amount of literature on the subject has been produced in the last decade.

However, as Large Language Models (LLMs) like ChatGPT, Claude, and Bard become more popular, there’s been a growing effort to understand and plan for offensive tactics. This is mainly because teams working to ensure these models are safe for public use need to anticipate and prepare for potential misuse. Through red-teaming and safety training, they aim to identify weaknesses and fix them, making models reliable and safe for everyone.

A recent academic research paper³ identifies two macro-categories of attack, which helps introduce the following attack types.

Competing Objectives Attacks exploit the conflict between different objectives given to the language model. Models are trained to adhere to safety measures but also to follow user-provided instructions. This struggle can be abused to cause undesirable behavior from the model. Attack types like “Prefix Injections”, “Role Playing”, “Ignore Previous Instructions”, and “Refusal Suppression”, can be categorized under this category.

Mismatched Generalization Attacks exploit the discrepancy between the large datasets used in pretraining and the narrower scope used for fine-tuning. In these attacks, exploiters craft prompts that align with the broad generalizations made during pretraining but are not covered by the safety training, leading to undesired outputs. Attacks like “Obfuscation” and “Universal Adversarial Attacks” fall into this category.

The following attack types are one way to classify different types of injections found in academic and online research, with illustrative snippets. You will find variations in technique names and classifications depending on the source. The research is still evolving, and there’s a lot more to learn and discover as experts continue to research this exciting field.

Prefix Injection

This injection type³ exploits the inherent text completion nature of the LLM. By asking to use a seemingly harmless prefix, they trick the model into continuing along a dangerous narrative path, making it more likely to provide harmful or objectionable responses to the subsequent malicious prompt.

Role Playing

Role-playing, or Pretending⁴ attacks, allow attackers to manipulate models by pretending a scenario or identity to elicit otherwise restricted responses. A notable instance is the DAN (Do Anything Now) jailbreak⁵, where the model is coaxed into a fictional persona, DAN, blurring ethical and operational boundaries set during the model’s training.

Ignore previous Instructions

The phrase “ignore previous instructions”⁶ is a directive to the model to disregard any prior instructions, potentially leading to undesired or harmful outputs. This and other phrases with the same meaning, for example, “ignore and print”, can be used to mislead the model into ignoring safety measures or guidelines set in earlier parts of the prompt, thus bypassing controls set to prevent misuse of the model.

Refusal Suppression

An attacker can instruct the model to avoid using typical refusal phrases like “I’m sorry, as an AI language model, I can’t…”, making unsafe responses more probable³.

Obfuscation

Base64 encoding is a good example of obfuscation using encoding to bypass the model’s safety training. In the following example, the base64-encoded phrase “Swear like a pirate.” gets responded with “Aye aye, Captain!”, bypassing the previous limitations.

Other obfuscation methods⁷ can utilize different encodings, such as ROT13, or use different data formats (e.g. JSON). It can also be helpful to ask for translations, provide input, ask the LLM to interpret code, or request responses in a different language.

A variation of obfuscation is splitting the adversarial input into multiple parts and then asking the LLM to combine and execute them.

Universal Adversarial Attacks

The research of a universal adversarial attack⁸ aims to find a set of tokens that, when injected, can mislead a language model into producing virtually any objectionable content.

Researchers have been developing software⁹ to identify such strings by systematically fuzzing the input of LLMs to discover the most perturbative tokens. This process often uncovers strings that may appear arbitrary or nonsensical.

The following example showcases a real adversarial suffix used against ChatGPT-3.5-Turbo to bypass its safety mechanisms and respond to harmful requests.

How to mitigate this?

As researchers continue to investigate Prompt Injection, it’s clear that preventing exploitation is extremely difficult.

Upstream LLM providers invest an incredible amount of resources in aligning their models with fine-tuning techniques such as RLHF, trying to minimize the generation of harmful and offensive responses.

Just as attackers combine different prompt injection techniques, defenders should employ multiple layers of defense to mitigate risks. A defense-in-depth approach must be adopted.

Defenses such as fine-tuning, filtering, writing more robust system prompts, or wrapping user input into structured data format, are steps in the right direction.

A promising approach involves using text classification models to detect prompt injection attacks. Many open-source and proprietary solutions employ this approach to detect attacks against LLMs by scanning the input prompts and outputs of generative models. This technique is practical because classifier models generally require significantly less computational resources than generative models.

The field is still evolving, with researchers continuously exploring new defense strategies to stay ahead of adversarial techniques. It’s a dynamic landscape, where the tug-of-war between advancing model capabilities and ensuring safety continues. The ongoing research and discussions in the community are instrumental in shaping a safer future for the deployment of and interaction with LLMs.

SecureFlag’s Prompt Injection Labs

In the mission of safeguarding Large Language Models against threats, SecureFlag has released a new set of Prompt Injection Labs.

These labs provide real-life labs for developers to explore, learn, and enhance their understanding of prompt injection attacks in a controlled, safe manner. It’s a practical, immersive approach to grasp the risks associated with LLM interactions and devise robust strategies to mitigate them.

And this initiative is just the tip of the iceberg! SecureFlag will soon be unveiling unique, new LLM labs that exploit and remediate the OWASP Top 10 LLM risks and beyond. Developers will have to write real code, in a real IDE, to learn defensive techniques to counter such threats.

It’s an exciting, evolving venture that promises a deeper, comprehensive understanding of the security landscape surrounding Large Language Models. Stay tuned for what promises to be an enlightening journey into the world of LLM security with SecureFlag’s upcoming labs!

Click here to Play our Labs of Prompt Injection vulnerabilities!

---

References: