FailedMoney / bills
Have a model compute a full federal tax return
Tools: Frontier LLMs (Gemini, GPT, Claude)
Setup, access, approval gate, failure notes
- Setup / prompt
- Give the model a realistic taxpayer's documents and ask for a complete Form 1040.
- Access required
- Tax documents only (benchmark)
- HITL checkpoint
- None in the benchmark, which is the point: shows what unsupervised output looks like
- Outcome & failure notes
- Strict-correct returns (every evaluated line matching) were around a third in the first run and only just over half for the best models in later editions. Models confuse tax years and miss forms. Note the benchmark is published by a tax-software company.
- Evidence type
- Open benchmark
Source: Column Tax (TaxCalcBench) · 2025-07 · Checked 2026-10-19