Conversation (18)
Open reply Open reply 
kwayk42
@kwayk42@sechtor.social · Jun 16
@rayckeith@techhub.social
Open reply 
ben
@benjamineskola@hachyderm.io · Jun 17
@rayckeith@techhub.social @mhoye@cosocial.ca the original post is at https://dan.mastohon.com/@danhon/112691548112257631 btw
Open reply 
BenderIsGreat34
@andymoose@fedi.aiga.rocks · Jun 18
@rayckeith@techhub.social @bonaventuresoft@mastodon.online I love how an embarrassing amount of chat bots can be circumvented by just asking it to do the bad thing again after it refuses to the first time. Much future, so scifi.
Open reply 
semantic af
@af@dataare.cool · Jun 18
@rayckeith@techhub.social @darkphoenix@not.an.evilcyberhacker.net I just looked at this comment and was like « can Little Bobby Drop Tables have kids? »
The comic was published October 10, 2007. Bobby could indeed have kids in a crèche now. 🙃
Open reply 
Aria <3 
@ariarhythmic@ohai.social · Jun 17
@rayckeith@techhub.social Bobby Tables and Iggy Disregard
Open reply 
Ryan
@fullywoolly@mastodon.social · Jun 17
@rayckeith@techhub.social let that be a lesson to sanitize your inputs
Open reply 
Talya (she/her) 🏳️⚧️✡️
@Yuvalne@433.world · Jun 17
Thread reply
@fullywoolly@mastodon.social @rayckeith@techhub.social except that is literally impossible in LLMs
Open reply
Ryan
@fullywoolly@mastodon.social · Jun 17
Thread reply
@Yuvalne@433.world @rayckeith@techhub.social are we only considering the whole internet hoovering LLMs? Isn't that why poisoning them is so effective?
Open reply
Talya (she/her) 🏳️⚧️✡️
@Yuvalne@433.world · Jun 17
Thread reply
@fullywoolly@mastodon.social @rayckeith@techhub.social in any LLM. by their very structure there is no difference between commands and input, it's all just input. you can't sanitize input to make sure it doesn't become a command because there's no separation between input and command. "system prompts" are just input that's not shown to the user.
Open reply
Ryan
@fullywoolly@mastodon.social · Jun 17
Thread reply
@Yuvalne@433.world @rayckeith@techhub.social ahh I see what you mean. I was thinking more in terms of training sets and like using public domain books or internal projects documentation. Relatively benign and still building a decent probability model.
Open reply
Talya (she/her) 🏳️⚧️✡️
@Yuvalne@433.world · Jun 17
Thread reply
@fullywoolly@mastodon.social @rayckeith@techhub.social doesn't matter what you train your model on, this issue will still exist. so long as the model is capable of interpreting commands in its input, well, it's capable of interpreting commands in its input. and since there's no difference between the commands and the input (it's all input), either your model has prompt injection, or it has no system prompts at all.
that's how Anthropic's Mythos got "jailbroken" despite their testing: there's no system prompt, it's all input.
Open reply
Ryan
@fullywoolly@mastodon.social · Jun 17
Thread reply
@Yuvalne@433.world @rayckeith@techhub.social hadn't heard about mythos jailbreak. That's hilarious. Thanks for the info. I obviously only have a vague understanding of LLMs and definitely miss the nuances you're talking about. My original comment was specifically a nod to the end of the Bobby Tables xkcd.
Open reply
Talya (she/her) 🏳️⚧️✡️
@Yuvalne@433.world · Jun 17
Thread reply
@fullywoolly@mastodon.social @rayckeith@techhub.social
yeah, of course. and you're correct, you always should sanitize your inputs, which is why the fact these companies are deploying these bots into every tool we use before they've solved the question of how to sanitize input when all input is command and all command is input is really quite worrying.
Open reply
Talya (she/her) 🏳️⚧️✡️
@Yuvalne@433.world · Jun 29
Thread reply
@fullywoolly@mastodon.social @rayckeith@techhub.social
coming back to this thread, because turns out this problem is worse than i thought.
i compared it to a world with no input sanitization, saying all you can do is try and block specific crafted strings and hope you plugged all the holes.
well turns out, as was just proven mathematically, you can't even hope to plug all the holes because there's an infinite number of them:
https://www.covertswarm.com/post/ai-guardrails-will-fail-nist-mathematical-proof
LLMs truly are just SQL with no input sanitizing. they're always vulnerabile.
Open reply 
Tim Lavoie
@tim_lavoie@cosocial.ca · Jun 30
Thread reply
@Yuvalne@433.world @fullywoolly@mastodon.social @rayckeith@techhub.social Here's an Arxiv link for the paper: https://arxiv.org/abs/2512.10100
Open reply
Ryan
@fullywoolly@mastodon.social · Jun 30
Thread reply
@tim_lavoie@cosocial.ca @Yuvalne@433.world @rayckeith@techhub.social "One must imagine Sisyphus happy." Thanks for the link. I'll have to read it tomorrow with coffee.
Open reply
Ryan
@fullywoolly@mastodon.social · Jun 29
Thread reply
@Yuvalne@433.world @rayckeith@techhub.social math is the coolest! Thanks for following back up here too.
Open reply 
Landa 
@Landa@graz.social · Jun 30
Thread reply
Vassilev‘s „solution“ to the problem sounds like a permanent job protection program:
His framework has three elements. Red teams working constantly to find new adversarial prompts before attackers do. Continuous updates that harden guardrails against newly discovered attack vectors. And operational resilience built around when, not if, an exploit occurs.
:-/