Post #1744537
2026-04-28 10:49 UTC
Replies (21)
-
@arclight@oldbytes.space 2026-04-28 13:35
Once upon a time - around 2007 or so, just before I left sysadminery to do risk assessment on legacy radioctive waste cleanup - I was upgrading a Blackboard LMS. Hardware load balancer in front of two unreliable web/app servers, Oracle db with RMAN backups, NFS file store backed by iSCSI SAN storage for user files. The web front-ends were intentionally provisioned with low disk because they didn't need it - content was on the NFS server or in the database and all the front-ends needed disk for was swap and log files from Apache and Tomcat. Those were religiously scraped because why would you expect the vendor to rotate their giant useless log files or record to a remote log host when they could just let crap accumulate everywhere until their system fell over? But I digress. I had automated log cleanup and database backup (with tested restores) and main IT managed the SAN backups. It was that brief Windows of low usage between semesters when we could run the vendor-supplied binaries to upgrade this expensive and cursed assemblage of Java, Perl, Oracle, and human misery. Read the documentation multiple times to understand the order of upgrade operations, clean and quiesce the system, take a few final backup snapshots and pull the trigger. The upgrade worked as intended, dutifully _moving_ files from the NFS mount of the large iSCSI drive to the web front-ends, filling the disk, then shitting the bed and falling over leaving the system in an unknown and unrecoverable state. As one does when you are Blackboard, the usurious vandal of LMS vendors. Surveying the flaming wreckage, I called Bb support to as for guidance. The support peon was impressed with my calm tone. I responded that being outwardly furious and losing my shit at them was unlikely to recover my system or improve any outcome I cared about. I did ask in my support ticket if this behavior from the update was documented and if there was any way I had missed a critical "do this to avoid incinerating prod" step in the upgrade process. A few hours later I got a response back from upper tier support that no, I had read and done everything correctly according to their documentation and that this whole fiasco could have been avoided by the use of an _intentionally undocumented_ option to the updater. Their words: _"intentionally undocumented"_. Why? Somebody might get confused by the explanation so it was omitted in the interest of ... clarity? I spent several harrowing hours waiting for the iSCSI restore to complete. I did my best to verify no user content was lost but to this day I don't know if we lost data. Deleting one symlink in the filesystem would have prevented this problem, a symlink that was required in a previous version of the code for the system to work properly (one actually described in the vendor documentation). We were a small university and did not have a full replica dev system to test the updater on. Why would we? We explicitly did not do development, we ran a vendor-supplied code in production. Dev systems were for developers which we were not. This wasn't a matter of a spinning up some virtuals in the cloud - we bought and managed real hardware and there was no way in hell we could justify doubling our hardware investment to have a test environment just to verify vendor supplied code worked as advertised. There wasn't a possibility of auditing the updater to detect that it would copy and delete the entirety of user-uploaded content, not without decompiling a big blob of Java. I exercised what diligence I could given the garbage state of the vendor's code and still got royally fucked over. I owned that. I informed my management chain of the situation and kept them updated with new developments and a revised ETA until the system was stabilized, recovered, and updated. That is how I practiced server ops for a decade (1998-2008). You did your dligence, said a prayer as you pushed the button, and you owned the outcome. No idea what current practice is. That was almost 20 years ago before devops, virtuals, clouds, and containers replaced real machines and dedicated sysadmins. I would like to believe that outlook and practice carried forward since then but I don't know - I left for greener, safer, more relaxing and fulfilling pastures helping package, transport, and store Cold War era uranium-metal-bearing radioactive sludge, moving it out of crumbling fuel pools at Hanford to interim storage elsewhere at Hanford. Then later projects for Dounreay, Sellafield, US commercial plants, Swedish interim waste storage at CLAB underneath Oskarshamn, fire PRA for plants in the US, Sweden, and Spain, and a whole lot of safety analysis code development and software QA. Somewhere in there I live-tweeted Fukushima melting and exploding. That's possibly the most direct act of nuclear safety I've performed with the goal of keeping people informed well enough to contextualize what was happening so they didn't panic and hurt themselves.
-
@hipsterelectron@circumstances.run 2026-04-28 10:50
@arclight i immediately assumed this was an excuse for an intelligence operation
-
@snowyfox@deadinsi.de 2026-04-28 10:52
.
-
@rightsprung@c.im 2026-04-28 10:56
@arclight 100%
-
@NewtonMark@eigenmagic.net 2026-04-28 11:13
@arclight @arcadiagt5 @cdarwin Headline from two weeks into the future: “Software startup PocketOS, running out of runway, concocts a story that convinces investors the company is collapsing before it’s burned through all its money. Founders exit with the balance.”
-
@technicalotter@glammr.us 2026-04-28 11:26
@arclight “own your work” Preach! Preach this far and wide! Be proud of the actual things you create and be responsible for them. It’ll make your work better and you a better human being.
-
@gkrnours@mastodon.gamedev.place 2026-04-28 11:47
@arclight according to the article, they gave an access token with full access to the tool, used the tool on staging and it had side-effect over prod. The CEO seems to say it wasn't possible to have properly scoped access token, which seems weird and a big red flag if true
-
@ajn142@infosec.exchange 2026-04-28 11:53
@arclight shhhhh with your reasonable takes, the more they blame the LLM the less trustworthy all LLMs seem 😉
-
@cocoaphony@mastodon.social 2026-04-28 13:28
@arclight I agree broadly, but we're seeing too many events where an agent actively circumvents protections and escalates privilege. Our mental model is often "junior admin," but at times they act more like an attacker. It's hard to build a permission model for an entity that needs insider access to function, but does, at times, "go rogue." These posts are helpful to remind us how easy and common it is for these tools to break containment. Better training isn't enough. We need better tools.
-
@foolishowl@social.coop 2026-04-28 14:42
@arclight It does cross my mind that one reason to have an off-site backup -- or just air-gapped? -- is that a broken process can't reach it.
-
@kyonshi@dice.camp 2026-04-28 21:44
@arclight no, but management wants you to forego your responsibilities to cut corners, but if this shit happens who's gonna be the one the shit sticks on? I can tell you it won't be management.
-
@WizardOfDocs@wandering.shop 2026-04-28 22:44
@arclight this is indeed "AI" working as intended and it's the one thing LLMs should do more of: punishing those who put faith in them
-
@cybervegan@autistics.life 2026-04-28 23:56
@arclight I think it's the classic case of management thinking that all the effort in planning changes is spent on working out which buttons to press, rather than applying specific domain knowledge, experience and researching and testing the process. Sure an LLM can generate a simulacra of a change plan, including a bunch of steps to achieve the goal, but without peer review, a risk assessment and full back out plan, you're just going on hope and a prayer. The most stupid thing about the article was that the guy asked the LLM to explain itself, as if it was capable of having done anything other than spat out a Markov chain style output of what looked like a likely answer. It did not review what it had done. It was not contrite in the face of a fuckup. It just rolled the dice again and spat out what the prompter wanted to hear. I can't believe people are so taken in but this rubbish. As you say, it's essentially lack of control and blind faith in flawed tools with no technical oversight. Pretty much akin to running a script you found on the internet in a live environment without reading through to see what it does. People who do this basically deserve what they get.
-
@hyc@mastodon.social 2026-04-29 01:00
@arclight whatever. At this point, if bigtech corps want to destroy themselves with AI, let them.
-
@vintprox@techhub.social 2026-04-29 07:02
@arclight Speak some sense into them
-
@thegaffer@hobbitwhispers.social 2026-04-29 12:11
@arclight Great perspective. At the end of the day, though, someone made a risk-based decision (whether informed or not) to employ the chosen solution, and the responsibility is primarily theirs. And that person is usually a manager and/or executive.
-
@mcnado@mstdn.social 2026-04-29 13:30
@arclight this 1000x
-
@mathaetaes@infosec.exchange 2026-04-29 17:07
@arclight@oldbytes.space And another org learns Vanhoff's Law: "Anything that can be done automatically for you, will be done automatically to you."
-
@mlanger@mastodon.world 2026-04-29 17:09
@arclight@oldbytes.space Oh, I like the story as written. After all, isn't that one reason why people use AI systems? So they have someone else to blame without feeling guilty about it?
-
@canacar@ioc.exchange 2026-04-29 17:34
@arclight@oldbytes.space the problem is that it is not possible to "verify" those tools. Another part is, our current verification methods are deterministic. If a tool works correctly 90% of the time but has a 1% chance of doing something bad, there is no reliable way to verify this. The corrolary is that these undeterministic tools should.not be used. Yes, these things happen to people too, and yes, most of the damage could be mitigated by properly partitioning your environment, but if you use a tool to do X, and it has been working before, and suddenly it wipes the volume instead (with an API token it found in a file somewhere) because part of its training data contains instructions to wipe the volume before doing anything else, then I prefer to blame the companies that hype and push these tools and not the victims.
-
@nonehitwonder@tenforward.social 2026-04-30 18:48
@arclight@oldbytes.space "My AI went rogue!"