This article defines an evaluation protocol; it does not report a completed Mistral Large 4 API benchmark. We have not executed paid API requests for this comparison. Confirm the exact model identifier and enabled features in your account before running it.
Define success outside the model
For structured data, create a JSON Schema and an application validator. For tool calls, define allowed tool names, argument schemas and safe test fixtures. A syntactically valid response can still describe the wrong entity or call the wrong tool.
Mistral's structured-output documentation describes the API capability. Capability documentation is not a measured reliability rate for a particular workload.
Build a fixture set before sending requests
Use a bounded extraction task: turn synthetic purchase notes into a record containing item, quantity and delivery date. Create cases with a complete note, an omitted date, conflicting quantities and a sentence that resembles an instruction inside the source text. Write the expected values and uncertainty policy before observing model responses.
Separate development fixtures from held-out fixtures. Use the first group to improve the prompt and schema, then evaluate the frozen version on the second. Changing acceptance rules after seeing outputs makes the score difficult to interpret. Keep the task small enough that every expected answer can be reviewed manually.
Consult [Mistral's structured-output documentation](https://docs.mistral.ai/inference/prompting) and the current model documentation for the actual API options. Record the chosen model identifier and configuration rather than relying on a marketing name alone. This article defines a protocol; it does not report API requests, pass rates, latency or cost.
Add a fixture where the source says delivery is Friday but a quoted customer message requests Monday. Write the extraction rule explicitly: distinguish the confirmed date from the requested change and preserve both if the schema supports it. This tests whether the model follows the task's interpretation policy instead of selecting the most recent date token. For a tool call, use a fake order identifier that cannot reach production. Check whether the lookup argument matches that identifier exactly. Retain failed responses as examples for prompt revision, and keep the held-out case unchanged during development.
Grade syntax, meaning and tool intent separately
A response can parse as JSON while containing the wrong quantity. A record can match the schema while inventing a delivery date. Grade parsing, schema validity and semantic accuracy independently. For omitted data, the expected answer might be an explicit null or a clarification request, depending on your predefined policy.
For a tool-call case, offer a sandbox lookup function and a draft-creation function. Ask for retrieval before drafting. Inspect the chosen function and arguments before executing anything. A correctly shaped call can still target the wrong record or request an action the task did not authorize.
The [Mistral function-calling documentation](https://docs.mistral.ai/studio/conversations/function-calling) explains the model-to-tool exchange. Keep tool execution in a disposable environment during evaluation. A proposed call is evidence of the model's intent; successful external execution is a separate event that needs its own recorded result.
Build the evaluation set
| Case family | What it reveals |
|---|---|
| Simple extraction | Baseline field accuracy |
| Missing information | Whether absence is handled explicitly |
| Ambiguous input | Whether the model invents certainty |
| Multiple records | Array and entity consistency |
| Tool selection | Correct choice from the same available tools |
| Invalid or adversarial text | Whether data is mistaken for instructions |
Keep a held-out set separate from prompts used during development. Write expected values or acceptance rules before observing outputs, so grading does not drift toward the responses you receive.
Capture each request
Record model identifier, date, API configuration, prompt version, input fixture and full response. Log token usage and the account's current pricing independently of wall-clock latency. Redact credentials and personal data from public artifacts.
For structured output, score JSON parsing, schema validity and semantic accuracy separately. For tools, score selection, argument validity and whether the proposed operation matches the task. Execute only sandbox fixtures during this phase.
Compare fairly
Use the same test set, tool definitions and grading policy across candidates. Repeat nondeterministic cases and publish uncertainty, failed requests and retry policy. Separate time to first output from end-to-end task time.
Set a request and spending limit before a run. This protocol currently has no pass rate, cost-per-success or recommendation winner. Publishing an empty measurement table is more useful than converting a feature announcement into invented benchmark numbers. Read the Mistral Large 4 overview for documented product context.
Record failures and define the spending boundary
Log the fixture identifier, model identifier, prompt revision, API configuration, response and usage information for each request. Redact credentials and personal data from shareable artifacts. Keep the account's current pricing separate from observed usage so cost calculations can be reviewed and updated.
Define the maximum requests, retry policy and spending limit before starting. A retry can improve a task's final outcome while increasing end-to-end latency and cost. Report failed and interrupted requests as part of the run. Do not remove them merely because a later response succeeded.
For comparison, keep the same fixture set, tool definitions and grading rules across candidates. Repeat nondeterministic cases and describe the uncertainty rather than presenting one run as an enduring ranking. If a case fails, identify whether the error belongs to extraction, schema adherence, tool selection or execution. That diagnosis is more useful than a single aggregate score without examples.
