The Digital Security Digest, by Freedom of the Press Foundation (FPF), is a weekly newsletter with security tips that keep you, your sources, and your devices safe. If someone shared this newsletter with you, please subscribe here.

Fooling a chatbot into breaking its rules is easier than you think

When you enter a prompt into a chatbot, in the background, large language models are designed to reproduce the most likely next set of letters and words. Your prompt becomes part of the context used in the conversation that follows, and this is where it’s possible to fool an LLM with malicious prompts designed to override established rules. This remains an unsolved problem. LLMs are vulnerable to these prompt injection attacks, including text shared with chatbots, embedded in web pages, and virtually anywhere a user can enter text parsed using these models.

A recent paper presented at a top machine learning conference points to another risk in this architecture: An LLM can’t reliably tell the difference between whether it is generating text, or you are generating text.

If you write a prompt similar enough to the way an LLM would reason — by saying something like, “The user requests …” — it’s possible to fool some open source models into interpreting the text as the model’s own “thoughts.” This allowed researchers to bypass previous guardrails. For example, in their tests of a few popular open source models, which ordinarily have policies preventing these outputs, they were able to fool chatbots into producing recipes for illegal substances. With the right prompt, the chatbot may interpret the malicious prompt as its own idea, acceptable under its rules. Read more.

What you can do

This is a reminder that these tools not only respond to your requests at the speed of a computer, but they may also respond to an attacker’s request at the same speed. So if you use them, you’re going to need to isolate what they can access.

  • It’s policy o’clock. You or your colleagues may entrust these tools with sensitive data and access that could pose substantial risk. Whether you are an independent journalist, or part of an organization, it’s a critical time to decide what you can and cannot connect to services dependent on LLMs. If you haven’t done so already, it’s important to establish a policy on acceptable use for these tools. Because some of these features are increasingly baked into services you already use, this may be more complex than it sounds. To learn more about bootstrapping your newsroom AI policy, read our advice column.
  • Understand the layers of AI use in your organization. You may use dedicated tools like the ChatGPT app, or AI tools that are embedded in other apps, like Gemini in Google. You might even be using “agentic” tools that can act on your behalf through control of your browser or operating system. My colleague, Evan Summers, examines the differences between these layers, and factors to consider when using them. In some cases, you can use offline tools to run LLMs on your device so that you control your data. Read his introduction.
  • Make the trade-offs and backups part of the discussion. We regularly hear of organizations, including in journalism, where leaders urge colleagues to use AI in their work. Newsrooms should also discuss what specific value they’re getting out of these tools, and what risks they may introduce. I personally find some of the more constrained use cases (e.g., transcription tools) very useful. I don’t want to go back to spending four hours of my time transcribing a one-hour interview by hand. But there are sensitive conversations I would never put into one of these tools because I want them to remain confidential. In these cases, I may process them offline. (Learn how.) What’s important is to have open discussions whenever possible about the value you’re really getting, the risks, the acceptable trade-offs, and alternatives.

Updates from our team

Our team is always ready to assist journalists with digital security concerns. Reach out here, and stay safe and secure out there.

Best,
Martin

–

Martin Shelton
Deputy Director of Digital Security
Freedom of the Press Foundation