Why LLM Evaluation Scores Change Between acc and acc_norm

acc and acc_norm are both accuracy results, but they can select different multiple-choice answers because they compare candidates using different likelihood-scoring rules. Ordinary sequence likelihood sums token log-probabilities, which tends to penalize longer answers; acc_norm adjusts the candidate score for answer length. That adjustment can reduce one length effect, but it can also overcorrect. To interpret either score, check how the benchmark formats and scores its answer choices.

What acc and acc_norm measure

In likelihood-based multiple-choice evaluation, a model is scored by comparing candidate answers and selecting the one with the strongest score. Accuracy is then the fraction of benchmark questions for which the selected answer is correct. The distinction between acc and acc_norm is the candidate-scoring rule used before that fraction is calculated.

#	Preview	Product	Price
1		Practical LLM Evaluation for Production Systems: Measure, monitor, and improve AI system reliability...	$49.99	Buy on Amazon
2		LLMs in Production: From language models to successful products	$48.13	Buy on Amazon
3		Reinforcement Learning from Human Feedback: LLM alignment and post-training	$59.99	Buy on Amazon
4		LLM Evaluation at Scale: Testing, Regression Detection, and Quality Measurement for Generative AI...	$0.99	Buy on Amazon
5		Large Language Models in Finance: A hands-on guide to LLM architectures, agents, RAG, governance,...	$49.99	Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

• acc uses the task’s ordinary candidate-scoring rule, commonly summed token log-probabilities for a candidate sequence.
• acc_norm compares candidates after their scores have been adjusted for answer length. In real evaluations, the relevant length is usually measured in tokens, not characters.

The exact normalization is task- and implementation-dependent; the metric name alone does not specify every detail. In a Hugging Face Open LLM Leaderboard discussion, an organization representative described the normalization as being on answer length and noted that real evaluations usually use sequence tokens. Read the discussion.

Why answer length can change the selected choice

A sequence likelihood formed by summing token log-probabilities accumulates more terms for a longer candidate. In practice, this tends to make longer completions less competitive, even when their content is plausible. Length normalization modifies that comparison by accounting for the candidate’s length.

#1 Best Overall

Practical LLM Evaluation for Production Systems: Measure, monitor, and improve AI system reliability across training and inference

$49.99

Buy on Amazon

A simplified way to think about the difference is that ordinary scoring compares accumulated sequence scores, while normalized scoring compares scores adjusted for how long each answer is. This is an explanation of the mechanism, not a universal formula: evaluation implementations can differ in how they calculate or normalize candidate scores.

Option formatting matters

A benchmark might score a choice letter, such as “B,” or score the full answer text. Those are different candidate sequences with different token lengths and probabilities. As the Hugging Face discussion points out, score variation depends on how options are passed to the model. Consequently, two evaluations using the same model and benchmark name may not be comparable if they serialize choices differently.

Rank #2

Sale

LLMs in Production: From language models to successful products

$48.13

Buy on Amazon

Does acc_norm remove length bias?

No. Length normalization is a heuristic, not a guarantee of unbiased scoring. It can counter the tendency of summed likelihood to penalize longer completions, but it can also overcorrect and favor longer candidates. Which effect matters depends on the candidate-length distribution and evaluation conditions, including the benchmark, prompt, and few-shot setup.

There is no single universal percentage by which one metric is more biased than the other. A 2026 study examines how length bias varies across benchmarks and few-shot settings rather than establishing one number that applies to every evaluation.

Rank #3

Reinforcement Learning from Human Feedback: LLM alignment and post-training

$59.99

Buy on Amazon

Bayesian accuracy: a proposed alternative

Koen Oostermeijer’s paper, “Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative,” appeared in the Proceedings of the 43rd International Conference on Machine Learning in 2026 (PMLR 306, pages 94698–94713). It proposes Bayesian accuracy, which calculates posterior candidate probabilities using an explicit prior over answer length. The paper presents it as a drop-in alternative to likelihood-based multiple-choice evaluation that requires no additional forward passes, and reports lower empirical length bias than standard and length-normalized accuracy across the benchmarks and few-shot settings studied. Those findings describe the paper’s evaluations; they do not establish that the method resolves length effects in every task. Read the paper at PMLR.

What to check before comparing reported scores

A score comparison is meaningful only when the evaluation setup and reported metric are clear. Record or verify the following:

Rank #4

LLM Evaluation at Scale: Testing, Regression Detection, and Quality Measurement for Generative AI Systems

$0.99

Buy on Amazon

• Benchmark and split: Identify the exact task and data split.
• Prompt and few-shot setup: Note the prompt format and number of examples used.
• Candidate serialization: Establish whether the model scored choice letters or full answer strings.
• Scoring implementation: Check whether scores are summed likelihoods or length-normalized, and how normalization is performed.
• Model and evaluation versions: Record the model revision and evaluation harness or version.
• Metric definition: Consult the model report or technical report rather than inferring details from the label alone.

These checks matter in practice. Hugging Face’s archived Open LLM Leaderboard v1 documentation says it used the EleutherAI Language Model Evaluation Harness, selected metrics by benchmark, and that results could vary slightly with batch size because of padding. The v1 leaderboard was archived in June 2024, so this is historical setup documentation—not a description of a current leaderboard. See the archived v1 documentation.

How to read the labels

Treat acc and acc_norm as results under distinct candidate-scoring rules, not as interchangeable names for a single neutral measure. For a comparison, first confirm the task configuration and metric implementation; then consider how each rule interacts with that task’s answer lengths. A higher value under one label does not, by itself, show that the model is better overall or that the score is free from length effects.

Quick Recap

Bestseller No. 1

Practical LLM Evaluation for Production Systems: Measure, monitor, and improve AI system reliability across training and inference

$49.99

SaleBestseller No. 2

LLMs in Production: From language models to successful products

$48.13

Bestseller No. 3

Reinforcement Learning from Human Feedback: LLM alignment and post-training

$59.99

Bestseller No. 4

LLM Evaluation at Scale: Testing, Regression Detection, and Quality Measurement for Generative AI Systems

$0.99

SaleBestseller No. 5

Large Language Models in Finance: A hands-on guide to LLM architectures, agents, RAG, governance, and evaluation in finance

$49.99

Best Value

Sale

Large Language Models in Finance: A hands-on guide to LLM architectures, agents, RAG, governance, and evaluation in finance

$49.99

Buy on Amazon

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.