Post #1748420

2026-04-27 09:46 UTC

“begging stochastic systems to be good boys is not a security strategy” exhibit ♾️ What really grinds my gears is how a whole industry chose to forget the 101 of ops security. This is not an “AI&quot; problem, this is gross negligence by cloud providers like Raleway pretending &quot;having evals&quot; is an acceptable defense against catastrophic data loss &amp; users having production tokens of any kind lying around such that an attacker (or misguided agent) can find it. https://archive.ph/T3LU6

Replies (7)

@hynek@mastodon.social 2026-04-27 09:56

like the “AI" part is just a smoke screen to do something really stupid and point at all kinds of advertising materials and pretending to be the victim. and looking at the “wah wah we used flagship wah wah"… I'm having a gut feeling that the smarter the models become, the better they will get at a) convincing themselves any means are justified and b) finding adversarial ways to achieve whatever they want excitedly awaiting a report of how claude code used a kernel exploit to drop a database

Open ##2337909
@tdelmas@mamot.fr 2026-04-27 10:21

@hynek@mastodon.social Impossible to see the message without an X account.

Open ##2337911
@benjamineskola@hachyderm.io 2026-04-27 11:23

@hynek@mastodon.social “it confessed” is such a failure of critical thought, too.

Open ##2337916
@leathekd@hachyderm.io 2026-04-27 11:30

@hynek@mastodon.social What was striking to me was the seeming lack of personal accountability on the part of the author? The latest DB backup was 3 months old and the only other recovery strategy was volume backups on the same volume(?). Hindsight is 20/20 and all, but… offsite backups have been a thing forever. There is general awareness in the industry that LLMs will delete your computer on a whim and that they should be treated as polite but possibly insane junior developers with admin access. Or better still, don’t trust them at all and don’t deskill yourself while the LLM paints you into a corner.

Open ##2337917
@dtwx@mastodon.social 2026-04-27 12:51

@hynek@mastodon.social "What needs to change" well, you should probably stop using "AI"...

Open ##2337919
@robinsyl@meow.social 2026-04-27 13:40

@hynek@mastodon.social Honestly confused at the idea that an API endpoint should ask for confirmation for deletion. That's what a CLI is for. The API should just do as told. Deleting the backups when a volume is deleted is also strange. Honestly the entire situation is strange

Open ##2337920
@uncanny_static@chaos.social 2026-04-27 14:29

@hynek@mastodon.social Whenever I read that an "AI" "confessed" or "apologized for" actions it took, I want to tell the person making these statements that they are obviously not qualified to use such a system. 😵‍💫

Open ##2337922