AI takes on the calculations working physicists actually do
- What it is
- A benchmark asking AI agents to solve 50 unpublished, research-level physics calculations with checkable answers.
- Who did it
- A. Ramesh Chandra and collaborators, with specialist problem contributors.
- What it could mean
- Could reduce time spent on specialist calculations if the performance repeats beyond this test.
See the check plan
Evidence & validation
From announcement to evidence
A benchmark reports performance on a particular test. Read the problem selection, scoring method and limitations before treating a score as a scientific advance.
Read the original work
TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving ↗See the proposed checks
Rerun a bounded sample with released settings, repeat runs and compare answers and derivations.
Independent examination not started
Benchmark scores are not proof certification or confirmation of a new physical theory. The proposed checks remain work to do.
How we distinguish research evidence →
Calculations put to the test
The three physical motifs introduce specialist calculation tasks and answer comparison. The curves are invented illustration, not benchmark measurements or evidence that any derivation is correct.
Read the source · Abstract and evaluation setup · 50 tasks, final-answer grading ↗- Source kind
- AI calculation benchmark
- Evidence reported
- Numerical final-answer comparison; Released evaluation prompts
- AI role
- Agents attempted calculations with local tools and, on the realistic track, web search.
- Human role
- Experts supplied problems and reference answers; researchers designed the evaluation.
- Source date
- ; added 2026-10-15
- What it claims
- The authors report 49/50 correct final answers for GPT-6-Astra with Codex on 50 expert-supplied theoretical-physics calculations with tools available.
- Why this could matter
- AI takes on the calculations working physicists actually do Reliable calculation assistance could let researchers explore more questions. These tasks test extensions of known techniques, with one full run per agent and final-answer grading.
- If it holds up
- More researchers could obtain useful specialist calculations with less time and expense.
- If it does not
- Repeated tests would expose where final-answer accuracy overstates reliable research assistance.
- Impact horizon
- Methods · Theoretical physics · Research assistance
- Version
- arXiv v1
- Why we tracked it
- A recent benchmark of specialist calculations, with separate tool-enabled and web-disabled tracks.
- Highest-risk dependency
- One run per agent, possible literature memorization and answer-only grading limit generalization; this does not demonstrate breakthrough discovery.
- Available artifacts
- Manuscript, problem collection and evaluation configuration; no State of Proof reproduction executed.
- Current boundary
- Intake record only; examination not started. Source-reviewed benchmark intake; independent examination and reproduction have not started. A correct final answer does not validate every derivation.