AI tries open science—with an important judging caveat
- What it is
- A benchmark of 82 unresolved literature problems, with proposed answers assessed by four models.
- Who did it
- Zhiyi Li and collaborators working across mathematics and theoretical physics.
- What it could mean
- Could help select promising attempts for expert review if the models' judgments survive independent checks.
See the check plan
Evidence & validation
From announcement to evidence
A benchmark reports performance on a particular test. Read the problem selection, scoring method and limitations before treating a score as a scientific advance.
Read the original work
OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences ↗See the proposed checks
Select one judged success and one disputed case; inspect the decisive steps independently of the evaluator models.
Independent examination not started
Benchmark scores are not proof certification or confirmation of a new physical theory. The proposed checks remain work to do.
How we distinguish research evidence →
Proposed solutions need independent scrutiny
The paths and comparison lens introduce alternative attempts and their evaluation. No shape denotes a solved problem, measured score or certified proof.
Read the source · Abstract and evaluation · 82 problems and model judges ↗- Source kind
- Model-judged capability benchmark
- Evidence reported
- Model-generated evaluations; Reported solution-case comparisons
- AI role
- Models attempt open problems, and four evaluator models judge the proposed answers.
- Human role
- Researchers assembled literature problems and designed the evaluation scheme.
- Source date
- ; added 2026-10-15
- What it claims
- The authors assemble 82 unresolved mathematics and theoretical-physics problems, using four evaluator models; the highest reported mean model-judged solve rate is 14.0%.
- Why this could matter
- AI tries open science—with an important judging caveat A benchmark may reveal useful approaches to unresolved problems. Model agreement can share errors, so its judged solve rate is a hypothesis for independent review.
- If it holds up
- Researchers gain useful candidate approaches and a test bed for studying progress on unresolved questions.
- If it does not
- Independent checks would expose shared judging errors and improve how claimed progress is measured.
- Impact horizon
- Methods · Scientific evaluation · Open problems
- Version
- arXiv v1
- Why we tracked it
- An October benchmark makes the distinction between model judgments and independently checked solutions especially visible.
- Highest-risk dependency
- Evaluator models can share mistakes; judged success does not establish a solved problem or a correct proof.
- Available artifacts
- Versioned manuscript and benchmark materials; no independent solution verification executed.
- Current boundary
- Intake record only; examination not started. Benchmark intake only; independent examination has not started. The 82 problems are not 82 resolved discoveries, and model consensus is not proof.