Post #1722042
2026-04-19 17:06 UTC
Replies (13)
-
@whitequark@social.treehouse.systems 2026-04-27 01:17
@kirakira @jneen yesyesyesyes
-
@wbftw@hachyderm.io 2026-04-27 01:24
@kirakira @jneen yeah, had exact same conversation a few times too.
-
@AT1ST@mstdn.ca 2026-04-27 01:58
@kirakira @jneen This does remind me of, in a non-programming context, standardized testing [ https://youtu.be/J6lyURyVz7k?si=KW9Ub0JLAxazwQIq ]. Long story short, you can test for things well, but you can also test for things very poorly. As in "Pineapples lose races to hares because they don't have sleeves" bad.
-
@drwho@masto.hackers.town 2026-04-27 04:27
@kirakira @jneen They're to make Jenkins happy.
-
@Epic_Null@infosec.exchange 2026-04-27 14:04
@kirakira @jneen Hell, I often think the best tests I write are done in the middle of solving a problem when I have a middle piece that needs to be right but will be a pain to check as part of the whole. Oh, I need to handle a large range of values correctly, but it's not a simple lookup or math problem? That handling gets its own public function and you know I am testing every single one of them to ensure the values work out. (Okay that one time was because I had to build pretty terrible logic to make sure they all mapped correctly. Still. You get the point.) Oh, I need a system with potential edge cases? Isolate. Test. Odd math I don't understand, but that came from an expert? Isolate. Document purpose. Test. Document appology for terribly formed test. Document explination for terribly formed test. But these are middle-of-a-problem tasks that AI steals the discovery of. Cases where only familiarity of the logic tell you that you should expect issues.
-
@grumpasaurus@infosec.exchange 2026-04-27 17:18
@kirakira @jneen ai if anything is really raising up things that people really dont want to do and honestly don't care about.
-
@mensrea@freeradical.zone 2026-04-27 17:32
@kirakira @jneen to be fair, they don't kown how to use css or what ui is either
-
@lritter@mastodon.gamedev.place 2026-04-27 20:56
@kirakira @jneen they can't even write tests ;-)
-
@crowbriarhexe@tech.lgbt 2026-04-27 21:54
@kirakira @jneen oh no :(
-
@diazona@techhub.social 2026-04-28 02:45
@kirakira @jneen I recently engineered myself a fun story about asking an LLM to write a test suite for a script (a low-value helper script, so nothing serious being put at risk here) and it came up with a completely reasonable-looking test suite that mocked out networking routines in a completely wrong way, such that the tests did not in fact test anything close to what they were supposed. I spent several days chasing down that error. This was absolutely zero surprising, but it's fun to be like, hey, I made The Thing happen 😂
-
@pikesley@mastodon.me.uk 2026-04-28 07:01
@kirakira @jneen https://mastodon.me.uk/@pikesley/115059862333580759
-
@aj@home.ajacks.net 2026-04-28 07:23
@kirakira @jneen of nothing else, ai is at least showing clearly why software was already bad.
-
@alwayscurious@infosec.exchange 2026-05-08 18:29
@kirakira@furry.engineer @jneen@unstable.systems I used an LLM to write a test suite for some code I wrote myself. I also asked it to keep in mind the spec the code is supposed to implement. The LLM figured out how to achieve almost 100% MC/DC coverage using only the public API. It (correctly) justified condition that could not be tested. Some of the tests failed, and when they did, the LLM pointed out that the problem was with the code. In one case, it wrote a 2 line fix (add missing Vec::clears) I used directly. In one case, it suggested a 1 line change (use the correct Rust struct). In the others, I fixed the code myself. I didn’t do this because tests are unimportant. I did it because writing a test suite with good coverage by hand is very difficult. Is this a silver bullet? Of course not. I still need to cleanup and validate the test suite, and I need to write integration tests. But it saved me a lot of time, and resulted in much better test coverage. If nothing else, this makes regressions much less likely.