Post #1770963

I recently listened to the Well There's Your Problem episode about the Therac-25. It's from a few years ago, so it predates the current moment of mass psychosis in the software world. I immediately wondered what the new "AI"-enabled Therac-25 was going to be, and then I remembered that we already have at least one, and probably more that we don't even know about: https://www.reuters.com/investigations/ai-enters-operating-room-reports-arise-botched-surgeries-misidentified-body-2026-02-09/ (CW for medical errors and body horror). Re-reading this article after listening to the podcast is chilling because of the parallels: the lack of effective regulation, the repeated denials from the company that anything is wrong, the lawsuits... (two of the Therac-25 incidents [that we know about] led to lawsuits that were settled out of court). It may be years before we know the full extent of the harm caused by tools like this. Yes, there are valid use cases for machine learning in medicine, some of which are safe. Guiding a surgeon inside a patient's skull clearly isn't one of them. What can anyone do about this as a patient? Ask hospital staff whether and how machine learning models are used in the software they use? How can they possibly answer that? Do your own research? This is all proprietary tech, so good luck with that. That's if you're even in a position to do anything; you could be going into emergency surgery, unconscious. Do we need "NO SLOP" medical bracelets? See the first point. Horrifying. I don't have a solution, except to hope that the bubble bursts and all these clowns go under before their products make it to where I live.