Recently I've started to find some AI-generated PRs a waste of my time. The contributor might spend five minutes generating a patch, but I need an hour to review it and figure out whether it actually solves the problem. This is especially frustrating with good first issue, a label that is supposed to make it easier for newcomers to start contributing to a project. Automated scripts can also find issues through labels such as help wanted or even bug, so the same problem can show up under those labels too.
I think the original idea behind this label is pretty good. Maintainers pick some relatively small and clear issues, and contributors can use them to get familiar with the codebase and the contribution process. The change itself may only involve a few lines of code, but reading the related code and discussing the implementation are also part of the experience. With AI, it becomes much easier to skip these steps. Someone can take an issue description, ask AI to generate a solution, and submit a PR without really understanding what they are changing.
The problem is that reviewing the result still takes time. Even if the patch looks reasonable and comes with tests, I still need to check whether it follows the project's existing practices, whether similar code already handles the problem, and whether the tests actually cover the behavior we care about. If the contributor hasn't considered these things, I have to go through them myself and then explain what needs to change. A small diff doesn't necessarily mean a quick review, especially when I also need to find out how much the contributor understands about their own implementation.
My first reaction was to look at the person's other contributions on GitHub. If a PR seemed to be generated by AI and the author had submitted a large number of PRs in the past month, I would be inclined to close it immediately. From the maintainer's side, it is hard to justify spending an hour on something the contributor may not have spent much time checking. But the number of PRs alone doesn't tell me what they considered for this particular change, so I started thinking about whether there was a better way to judge it before going into a detailed review.
One approach I'm considering is to ask how they discovered the problem and what they considered before implementing the fix. I tried this in badges/shields#12304, a PR that proposed suppressing XML parser warnings and errors in the dynamic XML service. It included tests and an explanation of the intended behavior, but I wanted to know more about how the contributor arrived at the implementation. Finding a problem through the issue list is a normal way to start contributing. What interests me more is what happened after they found it: did they read the related code, look at similar implementations, or consider why the project handles things in a certain way?
In that PR, what I really wanted to hear was whether they had checked how other endpoints handled similar errors. My concern was that the implementation only solved the problem in the current file without taking the project's existing practices into account. This is also a problem with the context given to AI. If you only provide an issue description and the file that seems relevant, AI may produce a local fix that looks fine on its own. But someone still needs to check the rest of the project and decide whether that approach fits. If the contributor leaves that work to the maintainer, the PR has arrived too early.
I don't expect a first-time contributor to already know all of our conventions. They may miss something even after reading the related code, and that is something we can discuss during review. But I do want them to be able to explain their choices and work through the feedback. Asking these questions seems more useful than trying to determine whether every line was written by AI. The answers will be different for each issue, and they give me a better idea of whether the contributor has actually thought about the change. A polished PR description on its own doesn't give me that confidence.
What makes this even more tiring is that sometimes, after you write a careful review, the contributor just passes your feedback to AI and pastes the generated reply back into the discussion. After a few rounds, it starts to feel like you're having a conversation with AI, with the person who opened the PR forwarding messages between you. The replies may look thorough, but you still don't know how much the contributor understands, and another round of discussion may not get you any further. If even the explanation of the implementation is left to AI, asking more questions won't necessarily tell me whether the contributor has put thought into the work.
After thinking about this, I still feel that good first issue has become a little awkward. It makes small tasks easier to find, and AI makes patches for those tasks easier to produce, but neither of them reduces the maintainer's responsibility for the code that gets merged. For now, I will probably ask more about how a contributor approached the problem before spending too much time on the implementation. But that can only be a starting point; a detailed reply doesn't necessarily mean they understand their own changes. I'm willing to help someone get familiar with a project, but I also need them to put effort into understanding the code they submit. Otherwise, even a small issue can take more time to review than it would take to fix myself.