Compare translation models
Run matched translation trials, preserve their conditions, and compare quality, cost, and missing outcomes for a defined use case.
Two models can differ in ways that matter for one kind of content and disappear in another. A useful comparison makes the intended decision specific: which configuration preserves a product's claims in the languages it serves, at an acceptable cost and failure rate?
Define the decision before selecting examples #
Name the content class, target languages, and requirement. “Which configuration preserves our checkout promises in French?” can produce an actionable result. A model ranking without content or language boundaries cannot explain where the result applies.
Select ordinary and difficult examples representative of that decision. Depending on the content, include short interface strings, ambiguous terms, connected prose, placeholders, and consequential claims. Keep enough examples in each relevant language to examine it separately from the overall average.
Preserve source revisions, target locales, context, and reference translations where available. A reference is useful only when its status and intended meaning are known; an existing translation is not automatically ground truth. Resolve or label source ambiguity before treating one plausible interpretation as a model defect.
Keep a separate set of examples for final testing that will not be used to tune prompts or guidance. Record exclusions under the criterion. Removing an awkward example after seeing an unfavorable result changes the experiment; missing source evidence for an eligible case remains an evidence gap.
Preserve the conditions being compared #
Save the actual configuration text or a retrievable revision, rather than only a mutable localization engine name. For each candidate, retain model and provider identifiers, fallback order, glossary, rules, voice, context, format options, pipeline and review policy, and criterion definitions.
When the hypothesis concerns the model, keep those other conditions constant. When it concerns guidance, keep the model constant and change the relevant glossary or instruction. Comparing a bare model with a fully configured localization engine can answer an operational question, but the difference includes context as well as model behavior.
Use an isolated comparison setup where possible because shared guidance may affect other callers. Inspect the baseline, propose the exact configuration change, obtain the required authorization, and verify what was saved before running the trial. Agent delegation helps define those boundaries for assisted work.
Run matched trials and keep unsuccessful attempts #
Submit the same source revision and target locale to each explicitly identified configuration. Preserve exact output with the trial identity. Record the model actually used and any fallback: a successful fallback does not establish the requested model's translation performance.
Keep retries and missing outputs in the record. Quietly substituting eventual successes makes a less reliable configuration appear complete. Repeat a subset of matched trials to examine variability; pinning an identifier does not guarantee deterministic output.
An agent can run the same selected strings through two authorized configurations and assemble the results. Specify whether it may only read existing settings, submit translations, or also change shared configuration. Showing two outputs in one conversation does not by itself create persistent experiment history or a statistically valid A/B test.
Evaluate meaning under the same criteria #
Define the criteria before revealing which model produced each output when feasible. For a delivery claim, the deadline, business-day qualification, and commitment must survive. A fluent translation that preserves the number but weakens will arrive to may arrive has changed the claim.
Calibrate the evaluator with separately adjudicated examples, including acceptable paraphrases, known errors, and cases lacking evidence. Keep judge identity distinct from generator identity. A model preferring its own output is not independent validation.
Ask qualified reviewers to examine consequential disagreements without model labels where practical. Compare the actual passages and requirements before averages. Validate quality findings helps settle suspected defects, while measurement design defines eligibility, false alarms, missed errors, and answer boundaries.
Report quality, coverage, and cost together #
Keep a row for each selected source revision, locale, candidate, trial, actual output, evaluation, and known execution cost. Report applicable cases, completed evaluations, determinate answers, and work with no completed answer. Unknown applicability should remain visible rather than being silently excluded.
Calculate pass rate among eligible determinate answers, and report determinate coverage against the named eligible population. Completed indeterminate evaluations count as completed but supply no supported quality verdict. Successful result history alone cannot reconstruct every selected item that failed to reach evaluation.
Break down outcomes by language and content group before pooling. A high average can conceal a serious failure in a small but important group. State how the sample was selected; a set chosen for known failures cannot establish the general defect rate of ordinary traffic.
Use the same accounting boundary for cost and latency. Include recorded retries, fallback, and additional stages. Missing cost is unknown, and missing output is unfinished work. A request's duration differs from total workflow time with waits and delivery. Edit distance cannot establish human time saved. Costs explains these record boundaries.
Retain a decision that can be revisited #
The report should connect a bounded conclusion to its evidence: which candidate met which requirements on which content and languages, with what coverage and operational cost. If the results do not distinguish the candidates, retain the existing configuration or collect the missing evidence rather than manufacture a winner.
Preserve inputs, outputs, configuration revisions, criteria, trial identities, inspected findings, and exclusions. Radar historical evaluation replaces answers in its interval, so it cannot preserve both versions of a comparison.
Before an authorized production change, retain the baseline and a rollback plan. Recheck the held-out examples and verify that the application receives the intended output. Revisit the conclusion when content, languages, or requirements change; the result describes the tested conditions rather than a universal model recommendation.