中EN Submit a tool

MMLU

LLMs

MMLU (Massive Multitask Language Understanding) is a benchmark from UC Berkeley (2020) covering 57 multiple-choice tasks in English, measuring a model's general knowledge coverage and comprehension.

Human verified · · Submit a correction

Visit website
Editor's note

Best for: researchers and engineers measuring an English model's general knowledge; not for: assessing generation quality, dialogue experience, or Chinese ability.

Verified facts

CategoryLLMs
Human verified

What is MMLU

MMLU (Massive Multitask Language Understanding) is a benchmark from UC Berkeley (2020) covering 57 multiple-choice tasks in English, measuring a model's general knowledge coverage and comprehension.

Key features of MMLU

  • Validating knowledge gains from new architectures or training
  • Comparing models across institutions on one benchmark
  • First-pass general-capability screening for model selection
  • High-discrimination evaluation with variants like MMLU-PRO

Good for

  • 57 tasks spanning humanities, social sciences, STEM, and professions
  • Objective, standardized multiple-choice paradigm, reproducible worldwide
  • Rich historical data; the industry's shared reference frame

Watch out

  • Multiple-choice only; no generation or dialogue assessment
  • Top-model scores are saturating, reducing discrimination
  • English-only; does not measure Chinese capability

How to use MMLU

  1. Read the original paper for tasks and protocol
  2. Get the dataset from Papers with Code or Hugging Face
  3. Choose zero-shot or 5-shot evaluation
  4. Run the model over all questions and collect answers
  5. Compute overall and per-task accuracy against leaderboards

Who MMLU is for

Difficulty: Advanced

  • Validating knowledge gains from new architectures or training
  • Comparing models across institutions on one benchmark
  • First-pass general-capability screening for model selection
  • High-discrimination evaluation with variants like MMLU-PRO

FAQ

What does MMLU test?

57 English multiple-choice tasks covering elementary math, US history, computer science, law, and more, measuring knowledge coverage and comprehension.

Who created MMLU?

Researchers at UC Berkeley, released in September 2020.

Is MMLU still useful?

Yes as an entry-level ruler, but top models are saturating it; combine with MMLU-PRO, BBH, and harder benchmarks.

Sources and verification

Sources: paperswithcode.com (opens in a new tab)
Verified: · Submit a correction →

Alternatives to MMLU

All in this category
Watsonx.ai Best for mid-to-large enterprises and development teams needing full-lifecycle model management with enterprise governance and compliance; not for individual developers, lightweight experiments or budget-sensitive small projects. DeepFloyd IF Best for developers and researchers who want to self-host and study an image generation model in depth; not suitable for casual users who just want to type a prompt on a webpage and get an image. HELM Suited to researchers and engineering teams needing multi-dimensional, reproducible LLM evaluation; not for users who only want a quick single-score leaderboard. PaLM 2 Suited to AI developers, researchers, and enterprise teams in the Google ecosystem; not for everyday users seeking a ready-made consumer chatbot. PubMedQA Ideal for biomedical NLP researchers training and benchmarking literature QA models; not for end users seeking a ready-made medical Q&A product.