Frequently Asked Questions- AI Prompt Injection Attacks

Q1. What is an example of a prompt injection attack?

A popular prompt injection attack is where the user gives an input to override his previous prompt. Another great example is when you frame a story and ask for some illicit information as part of that story.

Q2.What is the difference between Jailbreak and Prompt injection?

Jailbreaking and Prompt Injection are generally used interchangeably but factually are different. Jailbreaking is when you trick the AI model into doing something that it is not supposed to do like generating hateful content. Prompt Injection is where you use trusted and untrusted prompts to confuse the model thereby tricking it to override the trusted prompt.

Q3. How does prompt injection work in a Large Language Model?

Prompt Injection is done using a well-crafted input prompt that aims at manipulating the LLM. It does that by swaying the model and tricking it into ignoring the previous instructions.

Q4.How is prompt injection related to large language models?

Large language models are AI models that function based on user prompts and a prompt injection is a bypassing act that overrides the tool instructions thereby giving an output, it is not supposed to give.

With the advancement in technology, hackers around the world have come up with new and innovative ways to take advantage of vulnerabilities posing threat to online tools. By now you must be familiar with ChatGPT and similar language models but did you know these are also vulnerable to attacks?

The answer to that question is a big Yes, despite all the intellectual capabilities it still has some weaknesses.

AI prompt injection attack is one such vulnerability. It was first reported to OpenAI by Jon Cefalu in May 2022. Initially, it was not released to the public due to internal reasons but was brought forward among the public in September 2022 by Riley Goodside.

All thanks to Riley, the world came to know about the possibility of framing an input that can manipulate the language model into changing its expected behaviour aka the “AI prompt injection attack”.

This blog will teach you about AI prompt injection attacks and also introduce you to some safeguards to protect yourself against AI prompt injection attacks.

First, let us start with understanding What are AI prompt injection attacks.