Post #4395036

A few thoughts on this new trend of AI reviews. For those that are unaware, if you submit a patchset to the linux kernel mailing list, as of recently, you will get ai reviews by a bot called sashiko. My experience as a developer with it has honestly been rather negative. Sashiko will generate way more comments than a human would, all pointing out potential problems with your patches. This has, for one, an impact on your morale. Your code is being absolutely shredded apart by a bot which isn't fun. On closer inspection though a lot of the comments are based on very shallow and inexact reasoning. They may look plausible and clever while being completely wrong and meaningless. So for one one must go through all of them and carefully judge each comment. Blindly applying the comments can lead to outright broken code, but at the same time among the comments there might be genuine problems being pointed out so ignoring them won't do either. The comments are not the result of proper reasoning but there's enough of reasoning like logic in them that it takes significant cognitive load to judge them. But then, if I'm submitting code to the kernel there might be things I'm unsure about, the kernel is huge and not every submitter will know the intricacies of every kernel API. Reviews generally play an important role there. If someone experienced points something out I can generally trust it and, even if the advice turns out wrong we can collectively take responsability for it and reason about it again when the bug shows up. But if the bot points out something about code I was dubious about, what should I do? Did human mantainers not point out the bug because the bot did and agreed with the comment? Did they just not notice either the bug or the comment? Should I just blindly trust potentially wrong advice? Should I ignore potentially good advice? If it goes wrong the bot can't be held responsible for it, only I can, so maybe ping human mantainers? But then on top of me having to do more work to sift through the huge volume of comments human reviewers need to spend more time carefully analizing those comments. In the end it adds more friction and more work for everyone involved, and it's non trivial. Arguably the end result might be better, among the many comments the bot makes there could be some genuine bug or security vuln that might have gone unnoticed. But at what cost? The cost is huge and I'm not even getting into the environmental implications and whatnot. As far as I'm concerned I'd rather go back to the old workflow where comments I got in my patch where all meaningful and an opportunity to learn. Merging bugs has always been a part of the game and trying to preemptively prevent them is infeasible anyway. What good is it to have a dubious and marginal improvement in the final code quality if the price to pay is making the process so much more exhausting and un-human for everyone? What impact will that exhaustion have on the code long term? It may very well be a net negative. If someone really cared about the potential bugs sashiko can catch they should run it on the codebase themselves and do the work of sifting through the noise out of choice. Even then, filtering without enough care can lead to needlessly burdening patchsets that mantainers have to review.