Note: this article has been revised from a previous version to use MBPP Plus instead of the original MBPP to reduce noise from ambiguous problem statements. It also has been updated to use lower-quality models to use models that had not saturated MBPP Plus
Background
With the adoption of agentic coding tools, the majority of code currently written has been written by AI. Usage varies by company, domain, and language, but Google and Anthropic put the share of new code generated by AI at their companies at 75% and 80% respectively. As more code is generated by LLMs, developing accurate LLM-led reviewing protocols will be essential to developing functional, maintainable code at high velocity.
One approach to developing LLM-led reviewing protocols is to try to increase the diversity of agents that are used in code development. In many code development organizations, having two separate humans review code seems to improve code quality, so having diverse LLM agents working on code seems like it should improve outcomes.
Diversity of perspectives seems to consistently deliver better outcomes across many domains. Theoretical work shows that a diverse set of problem-solvers provides better problem-solving abilitythan a group of the highest-capability problem-solvers. This theory seems to hold up in reality–when human problem-solvers are put into groups of varying cognitive perspectives, the group’s problem-solving ability increases. In machine learning, ensembling refers to the practice of aggregating results from many smaller independent classifiers in order to create more accurate scores or predictions overall.
It does seem that using ensembles of LLMs can deliver improved performance on many benchmarks over a single model alone. And, LLMs are better able to find security issues within code generated from different models than code from the same model. Additionally, models exhibit self-preference bias and prefer outputs provided by themselves, perhaps due to lower perplexity associated with outputs created from the same model, so models reviewing their own code may struggle to spot issues in the same way that they might when reviewing other code.
However, there are some factors that might lead to models being better reviewers of their own code than of others’ code. The coding styles of different models are unique enough that it is possible to reliably classify code generated by a given model. It may be that a model reviewing code with less familiar code style (because it was not generated in the reviewer’s typical style) may have difficulty identifying an error. In addition, models from different providers tend to show high co-failure rates on benchmarks so it could be that “diversity” gains by using different models may not be sufficient to tackle problems of sufficient difficulty.
In this analysis, I’ll look at the effects of two ways of injecting “diversity” into LLM code reviews in order to ensure that code is functional:
Providing a different “persona” to the system prompt to create a review under different model activation patterns
Within the same LLM model, using different prompts may shift models’ internal activations to produce different “thinking” patterns. “Task vectors” emerge within language models when in-context learning techniques are used to change model behavior. It may be that these internal activation changes could shift the way that models process tokens enough to get “fresh eyes” and give effective feedback. Injecting diversity within the same model by using persona updates seems like it would keep a lot of the benefits of diversity injection while avoiding some operational issues associated with using different models for reviewers/code generators.
Using a different model as the code reviewing model to provide additional diversity
Models from different providers were trained on different data and different training strategies, so it is natural to assume that they might make errors in different ways. At some level, reinforcement learning is probably being used to either generate training data or directly update model weights, and the rewards chosen for that reinforcement learning process will yield models that perform differently. I couldn’t find any publications from Anthropic, Google, or OpenAI about their exact process here but here’s something from DeepSeek.
Prior to running the experiment, I expected that both diversity-improving interventions would be associated with better reviewer performance. I expected that modifying the system prompt would provide a smaller benefit than using a different model.
Setup
To evaluate coding performance under varying review conditions, I used the the Mostly Basic Python Problems Plus dataset, which includes ~400 problem descriptions as well as corresponding unit tests to evaluate successful code generation. I used all problems in the benchmark.
I chose LLMs from three providers (OpenAI and Google) that seemed to be at similar price and code performance tiers:
ProviderModelOpenAIgpt-5.6-lunaGooglegemini-3.5-flash-lite
For all models, I turned on some form of thinking (OpenAI supported an “effort” specification which I set to “medium”, and Gemini supported a “medium” level in their thinking config).
For each model, I provided a “persona” to the system prompt. One “persona” was meant to be a cautious, thorough coder, and the other “persona” was meant to creatively solve problems with less regard for rules or conventions.
Each model engaged in the following activities:
Initial code generation based on the problem specification (this was done only under the cautious persona, for all three model providers).
Initial code review for bug identification (this was done for all combinations of persona/model)
Post-review bug fixing (this was done only if the persona/model combination identified a bug)
All code can be found in my github.
Metrics
I was primarily concerned with recall and post-reconciliation error rate. In practice, better recall (# of correctly identified bugs / # of total bugs) means that you actually identify issues and are likely to actually fix them whether using the model that identified the error, another model, or human intervention. Post-reconciliation error rate (% of code chunks with errors after the full bug identification and reconciliation process) aligns with real-world utility of a coding strategy; if a strategy yields a low post-reconciliation error rate the reviewer model can fix the error quickly without needing any external intervention or complexity.
Results
Overall, reviewers had a 12.1% recall rate at identifying bugs present. Here’s a confusion matrix:
Recall Rate and Diversity-inducing Interventions
First, I estimate how the different diversity interventions influence recall rate (i.e. identification of bugs)
Optimization terminated successfully.
Current function value: 0.607659
Iterations 5
Logit Regression Results
==============================================================================
Dep. Variable: detected No. Observations: 535
Model: Logit Df Residuals: 530
Method: MLE Df Model: 4
Date: Sat, 15 Aug 2026 Pseudo R-squ.: 0.009071
Time: 10:44:06 Log-Likelihood: -325.10
converged: True LL-Null: -328.07
Covariance Type: cluster LLR p-value: 0.2028
===================================================================================================
coef std err z P>|z| [0.025 0.975]
---------------------------------------------------------------------------------------------------
Intercept -0.6784 0.247 -2.752 0.006 -1.162 -0.195
C(generator_model)[T.gpt] -0.1721 0.250 -0.688 0.491 -0.662 0.318
C(reviewer_model)[T.gpt] 0.0480 0.200 0.241 0.810 -0.343 0.439
C(reviewer_persona)[T.Creative] 0.1741 0.074 2.351 0.019 0.029 0.319
concordant -0.3850 0.120 -3.208 0.001 -0.620 -0.150
===================================================================================================
The coefficient on model concordance was negative–model configurations where the reviewer was the same model as the code generator had lower rates of identifying true bugs than model configurations where the reviewer was a different model from the code generator.
The reviewer “persona” also affected code detection capability substantially, at about half the magnitude of using a different model. I would expect that for optimal bug-detection capabilities, finely tuned prompts as well as discordant model reviewers would produce the best results.
Here’s a graph showing how this breaks down by model provider
Post-Reconciliation Error Rate and Diversity-Inducing Interventions
Now, we’ll estimate the cumulative effect on code quality via the end-to-end pipeline from different diversity configurations:
Optimization terminated successfully.
Current function value: 0.511585
Iterations 5
Logit Regression Results
==============================================================================
Dep. Variable: post_error No. Observations: 3023
Model: Logit Df Residuals: 3018
Method: MLE Df Model: 4
Date: Sat, 15 Aug 2026 Pseudo R-squ.: 0.002202
Time: 10:44:07 Log-Likelihood: -1546.5
converged: True LL-Null: -1549.9
Covariance Type: cluster LLR p-value: 0.1454
===================================================================================================
coef std err z P>|z| [0.025 0.975]
---------------------------------------------------------------------------------------------------
Intercept -1.2257 0.121 -10.129 0.000 -1.463 -0.989
C(generator_model)[T.gpt] 0.0147 0.078 0.189 0.850 -0.138 0.167
C(reviewer_model)[T.gpt] -0.2334 0.058 -4.022 0.000 -0.347 -0.120
C(reviewer_persona)[T.Creative] 0.0071 0.024 0.297 0.767 -0.040 0.054
concordant -0.0061 0.038 -0.161 0.872 -0.080 0.068
===================================================================================================
Diversity interventions were not significant here–turns out that relatively few errors were identified, and it was very infrequent that code review changed the code state between working and non-working.
Conclusions
It does seem that the practice of having distinct generator/reviewer models may improve the reviewer’s ability to identify issues upon review.
The effect of creating different “personas” on the same model was too small to measure here. I think it’s possible that this is because my “persona” prompt wasn’t detailed or long enough to shift model activations meaningfully. It could be that if I included specific examples or more details it would have led to better performance on reviews. In addition, our test did not include any code generated by the “Creative” persona which limits our ability to draw solid conclusions (what if the “cautious” persona is just way better at reviewing code and that swamps the diversity effect?).
Note that there are many different strategies that I would imagine would have larger effects than having distinct generator/reviewer models or different model personas. I think the biggest one is test-driven development (make sure that you have good test cases in place that the model can use iteratively to ensure the code works before it’s deployed) but I also think that providing project-specific code review instructions and using better models (e.g. Opus instead of Haiku) will deliver positive effects on performance.
Limitations
My theory on writing these posts is:
Anything worth doing is worth doing poorly
I’m not forcing anyone to read this stuff (so don’t have to feel bad if it’s not comprehensive)
I’m just in this for the love of the game (so I don’t really have to worry if people think I’m not working comprehensively enough)
So the scope of this analysis was kept pretty small. But, I’ll list out some things that I would hope to further improve on/investigate sometime in the future:
My coding “harness” was very limited and models did not need to use tool calls to evaluate code generated by other models. I believe that as codebases grow larger, model-specific choices for naming conventions and code organization may provide higher benefits for using a same-model reviewer to review code
This test involved only code generation; at companies, most AI-written code must be merged into a larger codebase
I used really cheap models for this test to avoid saturating the benchmarks, but I’m pretty sure very little actual coding is done by Haiku or GPT 5.4 mini in the real world.
I would love to look more into the “persona” perspective–I think that if I put some more time in, I might be able to write better prompts and achieve some diversity benefits there.