中EN Submit a tool

PubMedQA

LLMs

PubMedQA is a biomedical research QA dataset that answers yes/no/maybe questions from literature abstracts, with 1,000 expert-labeled instances plus large unlabeled and artificially generated sets.

Human verified · · Submit a correction

Visit website
Editor's note

Ideal for biomedical NLP researchers training and benchmarking literature QA models; not for end users seeking a ready-made medical Q&A product.

Verified facts

CategoryLLMs
Human verified

What is PubMedQA

PubMedQA is a biomedical research QA dataset that answers yes/no/maybe questions from literature abstracts, with 1,000 expert-labeled instances plus large unlabeled and artificially generated sets.

Key features of PubMedQA

  • Training and evaluating biomedical literature QA models
  • Comparing model results on the PubMedQA leaderboard
  • Researching NLP methods that extract conclusions from abstracts
  • Supporting clinical decision support and medical education research

Good for

  • High-quality expert annotations with a crisp task definition
  • A sensible three-tier mix of labeled, unlabeled and generated data
  • A public leaderboard that eases comparison and iteration

Watch out

  • Answers restricted to yes/no/maybe, a narrow task format
  • Focused on English biomedical literature
  • Requires machine learning engineering skills to use effectively

How to use PubMedQA

  1. Clone the GitHub repo and download the dataset
  2. Load the data to understand question, context and label structure
  3. Preprocess and tokenize the data for your model
  4. Train with an architecture such as BERT or PubMedBERT
  5. Evaluate on the test set and submit via the Submission guide

Who PubMedQA is for

Difficulty: Intermediate

  • Training and evaluating biomedical literature QA models
  • Comparing model results on the PubMedQA leaderboard
  • Researching NLP methods that extract conclusions from abstracts
  • Supporting clinical decision support and medical education research

FAQ

What is the dataset composed of?

1,000 expert-labeled QA instances, 61,200 unlabeled instances and 211,300 artificially generated QA pairs.

What kind of questions does it answer?

Biomedical research questions in yes/no/maybe form, such as whether a treatment is effective, answered from literature abstracts.

How do I submit to the leaderboard?

Follow the Submission guide in the GitHub repository to upload predictions and metrics, then wait for review.

Sources and verification

Sources: pubmedqa.github.io (opens in a new tab)
Verified: · Submit a correction →

Alternatives to PubMedQA

All in this category
MMLU Best for: researchers and engineers measuring an English model's general knowledge; not for: assessing generation quality, dialogue experience, or Chinese ability. Watsonx.ai Best for mid-to-large enterprises and development teams needing full-lifecycle model management with enterprise governance and compliance; not for individual developers, lightweight experiments or budget-sensitive small projects. DeepFloyd IF Best for developers and researchers who want to self-host and study an image generation model in depth; not suitable for casual users who just want to type a prompt on a webpage and get an image. HELM Suited to researchers and engineering teams needing multi-dimensional, reproducible LLM evaluation; not for users who only want a quick single-score leaderboard. PaLM 2 Suited to AI developers, researchers, and enterprise teams in the Google ecosystem; not for everyday users seeking a ready-made consumer chatbot.