Post #3349402

2026-06-11 07:21 UTC

Yeah, LLMs are gonna spin their wheels hard when it comes to testing anything at the kernel/os level, if you dont have automated testing with a virtual machine setup to actually be able to replicate a bug, you 100% just cannot test anything they produce or say As soon as you have the ability to go “Okay we have a failing test, make it pass”, the LLMs get a lot less stupid, because instead of just randomly fumbling around and guessing, they have actual feedback to iterate on and can actually chew on it til they fix the issue or give up.

Replies (2)

@jj4211@lemmy.world 2026-06-12 17:58

Not just automated testing but, for CodeGen to really work 'agentic' like: - You need that automated test case to trigger the misbehavior 100% of the time (often, the act of figuring out how to trigger the misbehavior means you already know the fix, but not always) - That automated test case needs to be succinct and as much as possible, feed *only* the problematic output back to the CodeGen. CodeGen can easily get distracted by irrelevant input - That automated test needs to be very quick from time to code change to test case completion. Even with everything just right, expect the CodeGen to basically thrash around guessing things that sound right but to no avail. Most attempts summed up as: "Ok, the problem is absolutely caused by , and here is the definite fix and it is complete but just double checking... Ok, that didn't quite fully fix it... see next attempt. So a long test case can make it take an eternity as the CodeGen has to wait and run it over and over and over again, while a human might actually reason through it. - You need to let the token hose go. It's guessing and it can take quite a few guesses to get right. - Be prepared for pointless code changes along the way. It makes guesses and often leaves the wrong guesses in, doing nothing at all to help the problem, but potentially having side effects. It decides that while it didn't work, it must have been a part of the solution, and that it must be left in. - Consequently, you better have an amazing test suite to capture the likely side effects of those spurious changes, or be prepared to unwind the progress and extricate the result manually.

Open ##3351296
@schipelblorp@sh.itjust.works 2026-06-11 19:29

I'm not a programmer, but isn't reproducing a reported bug step 1?

Open ##3351297