Can we trust “done”?

The study at a glance

A completion evaluator can be wrong because the evidence it receives is incomplete or stale, even when the task's outcome criterion is valid. We present a reproducible stress-testing protocol that separates complete-state truth from exposed evidence and retains success, failure, and unknown judgments. Standard precision, recall, F1, coverage, and selective error describe both incorrect decisions and successes left unrecognized. The artifact covers 20 fault families in three emulators, 31 HTTP/SQLite families with retries and crashes, and eight interventions on 91 externally authored retail tasks. A closure-and-revision gate reduces false admissions but lowers service recall from 86.7% to 46.7%. We then evaluate bounded additional-evidence requests on 620 paired cases spanning four collection conditions. When collection recovers, full and targeted requests both restore recall to 100.0%; targeted requests use 135 additional provider reads versus 150 for full recollection. Across all conditions, targeted requests achieve 83.0% precision and 65.0% recall, while an acquisition-capable baseline achieves 78.9% and 75.0%. Outages, continuing revision changes, and dishonest or incomplete collection retain important failure modes. The contribution is an executable evaluation protocol and a measured decision/request interface, with explicit acquisition costs and trust limits.

A completion claim needs enough evidence.

An agent can report success while the evidence is incomplete or out of date. Requiring stronger proof can reduce false success claims, but it can also leave genuinely completed tasks unrecognized.

We evaluate that tradeoff and test whether a bounded request for specific additional evidence can resolve uncertainty without repeating every check. The framework scores success, failure, and unknown judgments against independently checked task state.

Stricter checks trade recognition for assurance.

In the service study, a gate that checks history completeness and resource revisions reduced false success admissions, while completion recall fell from 86.7% to 46.7%. Here, recall measures how many successful tasks the evaluator recognized.

When evidence collection recovered, both full and targeted recollection restored recall to 100.0%. Targeted requests used 135 additional provider reads, compared with 150 for full recollection.

Those recovery results do not hold across every condition. Across the constructed request-study mix, targeted requests achieved 83.0% precision and 65.0% recall. Outages, continuing changes, and incomplete or dishonest collection retained failure modes.

These are results for controlled experimental conditions, not estimates of production reliability. The paper reports comparisons, costs, and counterexamples.

What the experiments cover

• 20 fault families across refund, deployment, and export emulators.
• 31 service families with persistent HTTP/SQLite state, retries, and crashes.
• 91 externally authored retail tasks replayed under eight evidence interventions.
• 620 paired request-study cases spanning four collection conditions.

The current study contains no live LLM-generated agent trajectories, customer workloads, or commercial provider transactions. External task replay broadens the test environment; it does not measure conversational agent performance.

The 620 cases reuse the service families under constructed conditions. They are not 620 independent task designs. Validation on real agent runs remains future work.

Download the full study.

Inspect the protocol, code, and results.

The public artifact includes task definitions, complete-state checks, evidence interventions, evaluator decisions, acquisition costs, and rerun instructions.

Completion-assessment research folder

The author is affiliated with AgentAction.dev and has a direct interest in the evaluated project. The paper discloses its use of AI-assisted tooling.

Help review this preprint.

We invite you to participate in peer review by submitting comments on the methods, findings, limitations, or related work. Reproduction reports and counterexamples are especially welcome.

Please identify the section or claim you are reviewing and include supporting evidence or a suggested correction where possible.

Submit review comments on GitHub

Comments are public. A GitHub account is required to submit a review comment.