Map

code-review-knowledge-transfer

Wiki conceptcode-reviewsoftware-engineeringempirical-research ↳ show in map Markdown
title
code-review-knowledge-transfer
type
concept
summary
Empirical finding that <15% of code review comments are about bugs β€” review's primary value is knowledge transfer, awareness, and alternative solutions, not defect detection
tags
code-review, software-engineering, empirical-research
created
2026-05-22
updated
2026-05-22

The dominant stated reason teams do code review is "to catch bugs." Multiple peer-reviewed studies and large industry datasets converge on the finding that this is wrong: bug detection is a minor fraction of what review actually produces. The primary value is knowledge transfer, team awareness, and surfacing alternative solutions.

This finding has structural consequences for any process β€” like blocking async pull requests β€” that is justified primarily on bug-catching grounds.

The studies

  • Microsoft Research, 2015 β€” "Code Reviews Do Not Find Bugs: How the Current Code Review Best Practice Slows Us Down" (IEEE). Only a very small percentage of review comments had anything to do with bugs. Most were about structural issues and style.
  • Bacchelli & Bird, 2013 β€” "Expectations, Outcomes, and Challenges of Modern Code Review" (ICSE). Studied review at Microsoft via observation, interviews, surveys. Less than 15% of issues discussed in code reviews relate directly to bugs. Finding defects remains the main stated motivation, but reviews are less about defects than expected. Primary outputs: knowledge transfer, team awareness, alternative solutions.
  • Bosu et al., 2015 β€” "Characteristics of Useful Code Reviews: An Empirical Study at Microsoft." 1.5 million review comments across five projects. More files in a change β†’ lower proportion of useful comments. Reviews over 20 files are already too big to review effectively.
  • Major tech company, ~9 million reviews internally β€” cited in Laforgia: knowledge transfer is the primary ROI. Up to 75% of comments affect software evolvability and maintainability, not functionality.
  • SmartBear/Cisco, 2006 β€” early case study; review's value already showed up in the maintainability column more than the defect column.

Why the gap

Reviews catch fewer bugs than expected because:

  • Most bugs that survive to the review stage are conceptual β€” wrong design, wrong invariant, missing case β€” not syntactic. Reviewers reading a diff don't have the context the author built over hours of work.
  • Reviewers are time-constrained and tend to comment on what they can quickly see: naming, formatting, structure, idiom.
  • The cheap-to-spot bugs are increasingly caught by static analysis, linters, and tests before review.
  • Review on large changes degrades β€” Microsoft's >20-file threshold means most PRs are already past the point where careful inspection is feasible.

Bacchelli & Bird's deeper finding: reviewers can't catch what they can't reconstruct. They lack the author's context, so they comment on the things context-free reading reveals β€” style, structure, naming β€” and miss the things context-bound reading would catch.

The implication for PR-as-default

This is the empirical hinge of stop-using-pull-requests. If review's primary value is knowledge transfer, then a blocking asynchronous queue is the worst possible delivery mechanism for that value:

  • Knowledge transfer is most effective when contextual and bidirectional. Async PR comments are neither β€” the author has moved on, the reviewer lacks context.
  • Knowledge transfer happens better via pair-programming (continuous, contextual), through walkthroughs, or via discussions during design β€” not after the code exists.
  • Defect detection, which would justify the blocking part, is precisely what review is bad at.

So the typical PR workflow optimizes for the wrong benefit using the wrong mechanism, and pays heavy waiting costs to do it.

What review still buys

The finding doesn't say "stop reviewing code." It says relocate review to where it produces knowledge transfer well:

  • Pair / ensemble programming β€” continuous review during creation. See pair-programming.
  • Non-blocking post-merge review β€” Thierry de Pauw's model in stop-using-pull-requests: reviews on mainline after merge, no blocking gate.
  • ship-show-ask β€” reserve blocking review for the small subset of work where the author actually wants gated feedback.
  • Design conversations before code β€” moves knowledge transfer earlier, when it shapes architecture rather than commenting on it post hoc.

Where the finding has limits

  • Reviews in regulated or safety-critical contexts (medical devices, finance, aerospace) serve audit and compliance roles that aren't captured by the bug-detection vs knowledge-transfer dichotomy.
  • Reviews on untrusted external contributions (open source from strangers, inner-source, semi-external teams) still serve a gatekeeper role β€” the original PR design point. See i-dont-want-your-prs for the LLM-era version.
  • The studies are heavily Microsoft-centric. Generalizing to OSS, very small teams, or non-software disciplines is reasonable but not directly evidenced.
Linked from