AI Code Review Mostly Misses the Point

Generative AI and large language models as embodied by ChatGPT are already having a big impact on how software engineers do their jobs. Their effectiveness shouldn't be surprising given that coding is a text-oriented medium.

I've seen suggestions and some early results around using ChatGPT for code reviews, with folks suggesting that ChatGPT makes great review comments, especially around style and maintenance issues. I'm sure some kind of “review copilot” will prove to be a useful tool. However:

It would be a big mistake to use AI as a complete replacement for human code reviewers.

To understand why, we should ask why we do code reviews at all.

First, let's get this out of the way: Code reviews do not find bugs. In fact, that's literally the title of this Microsoft Research publication. Sure, you'll occasionally turn one up, but bugs will always be a fairly small percentage of the identified issues.

And yes, code reviews are a line of defense against completely insane changes making it into the codebase. But hopefully you have a process in place so that insane things get caught far before the review stage.

One tangible benefit of effective code reviews is working the code into a long-term maintainable state, particularly with respect to style, language idioms, and design-in-the-small. This is where I expect AI to have the biggest near- and mid-term impact.

But what I've learned from working with large, high-performing engineering teams over periods of years is that

The single biggest benefit of code reviews is knowledge sharing and distribution.

Ensuring that at least two engineers (the “author plus n”) put thought into every single change pays great dividends in large and long-lived systems. AI probably won't save you if the author is unavailable when an incident occurs. But an additional human reviewer who looked at the code in question is invaluable. Additionally,

Code reviews are the single most efficient way to mentor new team members and junior engineers.

The benefits of having more experienced engineers review junior engineers’ code are obvious. Properly delivered, constructive review feedback can quickly help engineers level up early in their careers. Indeed, having new engineers jump in and start sending changes for review can be much more productive than weeks of “classroom” onboarding and training. What might be less obvious is that having new or less-experienced engineers review long-time engineers’ code is also a huge benefit to the team and the system, as they can bring a fresh perspective and challenge outdated assumptions.

What do you value most about code reviews? To what extent do you think they’ll be replaced by AI? Let me know, as my list here is far from comprehensive!