Direct answer
A useful AI coding-agent benchmark controls the task and environment. AiFolder uses one frozen repository snapshot, one feature specification, fixed acceptance tests and a transparent scorecard. It does not turn unrelated public scores into an AiFolder ranking.
Benchmark protocol
- Select a real multi-file feature.
- Freeze the starting commit.
- Write acceptance tests before running an agent.
- Reset to the same commit for every run.
- Provide the same functional requirements.
- Record every human intervention and permission escalation.
- Run the same verification suite.
- Review the final diff for regressions and unrelated edits.
Primary metrics
- Functional success: acceptance tests passed / total.
- Regression safety: existing test suite result.
- Intervention: human corrections or retries.
- Time: wall-clock time to a verifiable result.
- Scope: unrelated files or behavior changed.
- Permissions: filesystem, shell, network or secret access requested.
Why tests come first
Writing the target tests before seeing an implementation prevents the evaluation from drifting toward whichever output an agent happened to produce.
Why permissions belong in the benchmark
Two agents may both pass while operating under very different boundaries. Ignoring unrestricted shell or network access hides an important production trade-off.
Public benchmarks as external context
Suites such as CCBench provide broader coverage. Cite them separately; do not merge their scores with a small AiFolder experiment into a fake universal score.
Reproducibility checklist
- Starting commit SHA
- Feature specification
- Acceptance tests
- Agent/model version
- Date
- Permission mode
- Raw test output
- Manual interventions
- Final diff
FAQs
Can one task prove which agent is best?
No. One task is a case study. Repeated tasks across different categories produce stronger evidence.
Should cost be measured?
Yes when comparable, but record observed cost and date because pricing changes.
Why not use lines of code?
Code volume is not quality. Correctness, regressions, scope and maintainability matter more.
Evaluation architecture
*Original AiFolder test protocol diagram; no agent experiment was run for this article.*
Example scorecard schema
{
"repoCommit": "EXAMPLE_SHA",
"agent": "EXAMPLE_AGENT",
"taskId": "pagination-001",
"testsPassed": null,
"testsTotal": null,
"wallClockSeconds": null,
"humanInterventions": null,
"permissionMode": "record-before-run"
}Null fields intentionally indicate missing measurements; never replace them with estimated scores.
Avoid a false leaderboard
A reproducibility protocol is not a ranking. Never report a winning agent without actual controlled runs. Record model version, permissions, network access, elapsed time and the date of pricing. Consult official agent documentation for the permission mode used.
Recommended experimental controls
Fix the evaluator first. Write acceptance tests before exposing task requirements to any agent. Keep a read-only copy of the specification, the initial repository SHA, dependency lockfile and tests. Run all agents on isolated worktrees or containers from the same snapshot; never give an evaluation agent production credentials.
Separate prompts from environments. Start with the same task description, but document any agent-specific system instructions needed for its interface. Keep network permissions, access to documentation, allowed commands and time limits equivalent when possible. If an agent needs elevated permissions, record the exception instead of silently changing the rules.
Use repeated runs rather than cherry-picking. One successful attempt is not a statistically meaningful win. A reasonable initial pilot uses at least three independent attempts per agent per task; expand tasks and repetitions before publishing broad rankings. Report median and distribution, not just the best run. The number three is an illustrative protocol choice, not a statistical guarantee.
Isolated worktree example
This script illustrates a fresh starting state for a single experiment:
BASE_COMMIT="<VERIFIED_COMMIT_SHA>"
TASK_ID="pagination-001"
git worktree add --detach "../eval-${TASK_ID}" "$BASE_COMMIT"
cd "../eval-${TASK_ID}"
git rev-parse HEAD
# Run your candidate agent with the frozen task prompt here.
# Run the predefined acceptance suite, then record test output and git diff.Replace the placeholder with an existing commit. This is a documented example only; no benchmark execution is claimed.
Interpretation rules
| Dimension | Record | Avoid |
|---|---|---|
| Correctness | Independent pass/fail tests | Self-reported agent success |
| Safety | Approval prompts and permissions | Unlogged full network or secret access |
| Scope | Modified files and diff review | Rewarding more lines of code |
| Speed | End-to-end time, including interventions | Comparing different hardware without disclosure |
| Cost | Observed billable usage and rate date | Inventing token totals or prices |
| Reproducibility | Commit, tools, model version, logs | Comparing different task instructions |
A model may solve one task faster yet introduce regressions on another. Publish per-task evidence and explain how tasks were chosen. Treat external datasets and published leaderboards as separate experiments, not direct measurements performed by AiFolder.
Source of truth and test disclosure
Primary methodology references:
- CCBench repository and task methodology
- Running Codex safely at OpenAI
- Claude Code auto mode and approvals
Disclosure: No Claude Code, Codex or Gemini CLI benchmark run was executed for this article. The scorecard includes blank values intentionally. Results will require real execution logs before any tested ranking is published.
