Post #3743034

RE: https://infosec.exchange/@teriradichel/116839536008038553 A few core things that would help AI 🤖 models burn less tokens. 💰 Yesterday was all manual. The agents were just pounding on some things and not fixing certain bugs so I turned it off and dug into the code. So many errors were hidden by not pushing the bugs out of the subshell to the surface or not writing a message at all after an error occurs < and that is how AI models “fix” a lot of bugs. They don’t really fix the underlying problem they simply hide the error message. And in a very complex application like the one in my post below, that results in an application that appears like it’s working but it is skipping crucial bits. In my case it was failing to properly handle the resource already exists don’t redeploy it AWS Organizations delegated admins. When I bubbled up the error it stopped the app, which I’m doing to make sure I can find and fix all errors. It died on resource already exists errors. When I looked at the code for checking if a delegated admin exists it wrote separate code for every delegated admin unnecessarily and there were SO many bugs in that code. Each one was doing something different which would take a long time to fix and troubleshoot. I have already spent a lot of tokens on those bugs. I changed that code to use one helper method to handle the lookup so each delegated admin only sets a few variables and they call one helper. Why can’t AI figure out stuff like that? I explicitly define helpers and how I want them written in my README - sourced files not functions so I don’t lose error messages, though I may change that rule and add an explicit error handler to solve that problem instead. By fixing that problem manually I discovered my AWS command runner really should have a query option so I added it. That simplified the lookup. I use the command runner to validate all entries - like region. I also found and fixed issues in the AWS command runner. That core code was also hiding errors. The big one was fixing the eternal looping in my core parallel resource deployer. Hidden errors and core logic issues led to a number of issues getting this project done. I pulled the code into Google AI since it is not super proprietary and piece by piece asked it to find and fix problems. I had to ask multiple instances multiple times if they found issues in each other’s code and tell it not to change up the variable names (which Google AI likes to do with code snippets). Google AI was able to address the core looping problems and helped me unhide all errors and in theory fix some problems with the application traps. It also helped me add a core trap to stop the application on a certain code path - and to be honest I would not have thought of or known to add that trap. It’s the little tif bits like that which make AI very useful despite the painful errors. So now the app is deploying all resources, not hanging on prompts, and showing all errors. All the delegated admins are deploying correctly I think, though I need to login and verify that. I need to fix the tracker code so it has no AWS access and must only be set by the action files, so I think that will be moving to its own project. I may also move the parallel processor to its own project so that doesn’t get messed up again. I had configured a new account which wasn’t getting deployed and once I fixed all that it did. The configuration piece of this is working nicely though I did find a new bug. Mostly it is easy to add new account configurations and resources and in theory they just get deployed. These are the types of things you need to watch out for when writing code with AI. We would Huron a lot less tokens if someone could make it better at not hiding errors, not creating eternal loops, and not creating and therefore troubleshooting redundant code. As a reminder I wrote about the app I’m building in the post below.