Post #4377581

@arno_in_sing@mastodon.online @rgulick@social.coop Yes, absolutely, and there are examples of people doing this. Any time an LLM is exposed to untrusted data, assume an attacker can do anything that the LLM is authorised to do. Using one for spam filtering (aside from being laughably inefficient) is probably okay if the only output is 'spam' or 'not spam': the worst that can happen is that an attacker can make it treat their message as not spam, and that's possible with any spam filter, you're just making them do more work. The 'probably' is a problem if the LLM keeps state between messages, because a malicious message may be able to make it treat other, important, messages as spam. I don't like the term 'prompt injection' because it implies that normal operation of an LLM is doing something different. This isn't the case. The prompt isn't anything special for an LLM, it's just the first sequence of tokens it says. It is in no way privileged, anything that's processed after the prompt has the same ability as the prompt to influence the LLM's behaviour.