AI agents make dozens of tiny choices: which tool to call, whether to retry, which model to use, and when to escalate. Using a frontier LLM for every choice is convenient, but a new class of decision models asks whether routing can be smaller, faster and local.
Test status — October 2, 2026: this page defines AiFolder's independent benchmark. We have not filled the leaderboard with vendor numbers. Any published latency or accuracy claims must be reproduced before they become AiFolder results.
Why Strands Decider changes the Jev question
Our earlier Jev vs LLM benchmark plan asked whether a specialized decision primitive can replace generative classification. Strands Decider makes the question more interesting because a roughly 2B-class model is small enough to consider on-device and self-hosted routing.
The practical choice is no longer only “specialized API or LLM.” It can be local decision model vs hosted decision model vs full generative model.
The benchmark
| Workload | What it tests |
|---|---|
| Intent routing | Fixed multi-class decisions |
| Tool selection | Agent action choice |
| Support triage | Ambiguous labels and confidence |
| Model routing | Small-model vs frontier escalation |
| Policy checks | Boolean/enum reliability |
Metrics that matter
We will record macro-F1, Brier score/calibration, p50 and p95 latency, throughput, malformed-output rate, RAM/VRAM, cold start, energy where observable, and cost per 100,000 decisions. For agent routing, we also measure downstream task success because a cheap router that sends hard tasks to the wrong model can increase total cost.
Local AI test
The most useful experiment is privacy-sensitive routing on a local machine. The input set will contain synthetic support and code metadata, never private production data. We will compare a local Decider deployment with hosted decision APIs and a structured-output LLM while keeping labels and scoring identical.
Confidence is the product
A decision model is most valuable when its confidence is calibrated. A router should be able to say “I am uncertain” and escalate. We therefore treat calibration as a first-class metric rather than ranking systems by a single accuracy number.
When a full LLM still wins
Decision models are intentionally narrow. If the next step requires explanation, planning, synthesis or an open-ended action space, a generative model remains the natural fit. The architecture we expect to test is hierarchical: cheap constrained decisions around fewer expensive reasoning calls.
Reproducibility
AiFolder will publish task definitions, labels, model/runtime versions, warm-up policy and concurrency. Vendor benchmark claims stay attributed until reproduced. This lets the page become a living decision-model leaderboard rather than a launch recap.
