Amazon Researchers Propose Ising-Model Method to Correct Correlated LLM Judge Votes

Amazon scientists and academic collaborators have introduced a dependence-aware aggregation method using Ising models to adjust for correlated errors among large language model judges. Presented at ICML 2026, the unsupervised approach improved evaluation accuracy by 9% to 14% across relevance, toxicity, and summarization benchmarks compared to weighted majority voting, addressing redundant agreement caused by shared prompts and model lineages.

The Limitations of Majority Voting in Model Panels

Automated evaluation pipelines frequently deploy multiple large language models to judge outputs, such as scoring the relevance of passages retrieved by retrieval-augmented generation systems. Standard implementations aggregate these panel decisions through simple majority voting or weighted majority voting, where individual judges receive voting power proportional to their historical accuracy.

Both approaches rely on the statistical assumption of conditional independence, treating each judge’s errors as unrelated events. According to the research paper, titled Dependence-aware label aggregation for LLM-as-a-judge via Ising models, this assumption breaks down in production environments.

LLM judges frequently share training datasets, underlying model architectures, prompt structures, or phrasing sensitivities. When several models share these characteristics, they often replicate the same systematic errors. A panel where eight out of ten judges agree can present a false appearance of high confidence if those eight judges merely duplicate a single shared bias or rubric interpretation.

Network Modeling via the Ising Framework

To address correlated judgments, the researchers structured the panel as an interconnected network rather than a collection of independent scorers. The architecture uses an Ising model, a statistical physics framework designed to represent pairwise dependencies across collections of binary variables.

Through this framework, the aggregator estimates both individual judge reliability and the pairwise correlation between different judges simultaneously. The system operates entirely in an unsupervised setting, requiring no human-labeled ground truth for training:

• Latent Variable Estimation: The algorithm treats the true label of each evaluated item as an unobserved latent variable.
• Soft Probability Calculation: The system combines initial votes to compute a soft probability indicating the likelihood that an item’s true label is positive.
• Iterative Parameter Updates: The model alternates between refining these latent probabilities and recalculating individual judge reliability profiles alongside pairwise dependence scores.

The researchers developed two versions of the framework:

• Standard Model: Assumes the relationship pattern between judges remains consistent across both positive and negative labels. This produces an interpretable weighted vote where weights are mathematically adjusted to discount redundant agreement.
• Class-Dependent Model: Allows the correlation structure to change depending on the class label. This variant captures scenarios where judges agree broadly on clear-cut cases but divide into specific correlation clusters on ambiguous inputs, though it requires larger sample sizes to calibrate the additional parameters.

Benchmark Performance Across Evaluation Tasks

The research team evaluated the methodology across three binary classification tasks: relevance classification for retrieved information, toxicity classification, and summarization assessment.

The experimental setup utilized a panel of 10 judge models. All models operated at temperature zero to eliminate output variance from sampling randomness, ensuring that identical inputs produced deterministic responses. The dependence-aware models were benchmarked against uniform majority voting and weighted majority voting baselines.

The experiments produced the following accuracy outcomes using the complete 10-judge panel and full task datasets:

• Relevance Classification: The dependence-aware model achieved 0.912 accuracy, compared to 0.820 for weighted majority vote and 0.804 for uniform majority vote.
• Toxicity Classification: The dependence-aware model achieved 0.792 accuracy, compared to 0.694 for weighted majority vote and 0.695 for uniform majority vote.
• Summarization Assessment: The dependence-aware model achieved 0.806 accuracy, compared to 0.737 for weighted majority vote and 0.561 for uniform majority vote.

The dependence-aware variants consistently outperformed both baselines once panels contained enough evaluation logs to estimate pairwise relationships accurately.

Implementation Principles and Study Details

The study outlines several procedural adjustments for engineering teams operating multijudge evaluation systems. The authors recommend calculating panel-level statistical diversity rather than presuming diversity based solely on mixing model names or providers. They also advise inspecting agreement clusters in historical logs to uncover shared evaluation blind spots and adjusting confidence intervals to account for judge correlation.

The research was authored by Krishna Balasubramanian, an associate professor of statistics at the University of California, Davis, and an Amazon Scholar; Sasha Podkopaev, an applied scientist at Amazon Web Services; and co-author Shiva Prasad Kasiviswanathan. The complete findings were presented at the International Conference on Machine Learning (ICML) 2026.