Post #4377583
2026-08-04 13:16 UTC
Replies (2)
-
@david_chisnall@infosec.exchange 2026-08-04 14:35
@arno_in_sing@mastodon.online @rgulick@social.coop Yes, absolutely, and there are examples of people doing this. Any time an LLM is exposed to untrusted data, assume an attacker can do anything that the LLM is authorised to do. Using one for spam filtering (aside from being laughably inefficient) is probably okay if the only output is 'spam' or 'not spam': the worst that can happen is that an attacker can make it treat their message as not spam, and that's possible with any spam filter, you're just making them do more work. The 'probably' is a problem if the LLM keeps state between messages, because a malicious message may be able to make it treat other, important, messages as spam. I don't like the term 'prompt injection' because it implies that normal operation of an LLM is doing something different. This isn't the case. The prompt isn't anything special for an LLM, it's just the first sequence of tokens it says. It is in no way privileged, anything that's processed after the prompt has the same ability as the prompt to influence the LLM's behaviour.
-
@groxx@hachyderm.io 2026-08-04 14:36
@arno_in_sing@mastodon.online @rgulick@social.coop if you can control text that goes into an LLM, the answer is literally always "yes". not in the way that someone always eventually screws up and leaves a gap you can use, I mean *literally always*, there is no reliable defense at all. the only safety tactics that work are hard limits on what the LLM and its harness together can do. e.g. only output text that only humans will read -> your attack can't convince it to upload data elsewhere... but it might be able to trick someone into doing that for it, which sometimes happens with executable code snippets.