Why LLM Leaderboards Are Often Wrong
Small score gaps in model evals might just be noise. Learn how to use statistical error bars and rigor to determine if your model is actually better.
Best quote from Why LLM Leaderboards Are Often Wrong
The biggest red flag in AI right now isn't a low score—it’s a high score with no error bars. We need to stop treating evals like static scores and start treating them like the scientific experiments they actually are.
This audio lesson was created by a BeFreed community member
Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations Evan Miller Anthropic evanmiller@anthropic.com Abstract Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning. This article shows researchers with some training in statistics how to think about and analyze...
Frequently Asked Questions
Ranking models by tiny margins—such as a 0.5% difference—is often misleading because these fluctuations may simply be statistical noise rather than a reflection of true capability. Evaluation datasets are finite samples pulled from a theoretical "super-population" of all possible questions. Without calculating error bars or standard error, it is impossible to know if a higher score is a significant result or if the ranks would flip if the experiment were run again with different questions or different model seeds.
The Rule of Three is a statistical guideline used when a model passes every single test in a small sample size. If you run 30 safety tests and the model never fails, it is mathematically incorrect to claim the model is 100% safe. Instead, the rule dictates that the 95% confidence upper bound for the failure rate is 3 divided by the number of tests. In a 30-test scenario, you can only say with 95% confidence that the failure rate is below 10% in the wild.
Standard statistical assumptions require that every question in a dataset be independent, but real-world benchmarks often violate this by using multiple questions based on the same document or translating the same prompt into different languages. If a model struggles with the underlying context, it will likely fail all related questions, meaning they are not independent "votes" on performance. Clustered Standard Errors account for this correlation by grouping related items, preventing researchers from underestimating uncertainty and reporting artificially small error bars.
One of the most effective ways to shrink error bars is to use continuous metrics like "logprobs" (log probabilities) instead of binary pass/fail scores. By looking at the probability the model assigned to the correct answer rather than whether it happened to sample that answer, you eliminate "within-question" variance caused by the model's internal randomness. Other strategies include resampling (averaging multiple completions for the same prompt) and averaging results across the final few checkpoints of a training run to smooth out lucky fluctuations in model weights.
Comparing two separate error bars is often too conservative; models can have overlapping confidence intervals and still show a statistically significant difference. A paired difference test evaluates both models on the exact same set of questions and focuses on the gap between their scores. Because models usually agree on which questions are difficult, their scores are positively correlated. Subtracting these correlated variables shrinks the variance of the difference, making the test much more sensitive and capable of detecting real improvements that a naive comparison would miss.
Discover more
Python programming for LLMs and evals
Python programming for LLMs and evals
As AI integration becomes standard, the ability to both build and critically evaluate models is a vital technical differentiator. This path is ideal for developers and data scientists looking to transition from general programming to specialized LLM engineering and rigorous model benchmarking.
I want to learn the fundamentals of LLMs
I want to learn the fundamentals of LLMs
Large Language Models are revolutionizing how we interact with technology and information. This learning plan provides essential knowledge for developers, AI enthusiasts, and professionals who want to understand LLM capabilities, limitations, and future potential, enabling them to make informed decisions about implementing and working with this transformative technology.
Master ML Research in LLMs, NLP & Quant Fin
Master ML Research in LLMs, NLP & Quant Fin
This comprehensive track bridges the gap between theoretical machine learning research and high-stakes applications in NLP and quantitative finance. It is ideal for aspiring researchers, data scientists, and quantitative analysts looking to master the architectures behind LLMs and algorithmic trading systems.
Neural Networks and LLM
Neural Networks and LLM
This learning plan is essential for developers and data scientists looking to transition from basic machine learning to state-of-the-art generative AI. It bridges the gap between theoretical mathematics and practical implementation, making it ideal for those who want to build or fine-tune their own large language models.
Read academic articles vs Facebook posts
Read academic articles vs Facebook posts
In an era where social media algorithms prioritize engagement over accuracy, the ability to distinguish credible research from misinformation is essential for making informed decisions. This learning plan is ideal for anyone tired of falling for misleading headlines and wanting to develop critical thinking skills to navigate the overwhelming amount of information online with confidence and discernment.
Math for Stats, Probability & ML
Math for Stats, Probability & ML
This learning plan bridges the gap between theoretical mathematics and practical implementation in data science and AI. It is ideal for aspiring data scientists or engineers who want to move beyond using libraries and truly understand the logic driving machine learning models.
Advance probability
Advance probability
This plan bridges the gap between basic chance and high-level statistical modeling. It is ideal for data scientists, analysts, and decision-makers looking to master uncertainty and predictive accuracy in professional environments.
ML Eng: Math, Biz, Polyglot & Soft Skills
ML Eng: Math, Biz, Polyglot & Soft Skills
This comprehensive path is designed for engineers looking to evolve into senior ML leaders by blending technical depth with business acumen. It bridges the gap between low-level mathematical implementation and high-level strategic influence, making it ideal for those aiming to drive real-world impact in the AI industry.
From Columbia University alumni built in San Francisco
"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."
"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."
"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."
"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."
"Reading used to feel like a chore. Now it’s just part of my lifestyle."
"Feels effortless compared to reading. I’ve finished 6 books this month already."
"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."
"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."
"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"
"It is great for me to learn something from the book without reading it."
"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."
"Makes me feel smarter every time before going to work"
From Columbia University alumni built in San Francisco
"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."
"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."
"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."
"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."
"Reading used to feel like a chore. Now it’s just part of my lifestyle."
"Feels effortless compared to reading. I’ve finished 6 books this month already."
"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."
"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."
"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"
"It is great for me to learn something from the book without reading it."
"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."
"Makes me feel smarter every time before going to work"
"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."
"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."
"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."
"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."
"Reading used to feel like a chore. Now it’s just part of my lifestyle."
"Feels effortless compared to reading. I’ve finished 6 books this month already."
"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."
"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."
"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"
"It is great for me to learn something from the book without reading it."
"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."
"Makes me feel smarter every time before going to work"
"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."
"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."
"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."
"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."
"Reading used to feel like a chore. Now it’s just part of my lifestyle."
"Feels effortless compared to reading. I’ve finished 6 books this month already."
"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."
"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."
"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"
"It is great for me to learn something from the book without reading it."
"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."
"Makes me feel smarter every time before going to work"
