Design a translation-quality measurement

Turn a language requirement into a testable criterion, calibrate its answers, and report the frequency and scope of the problems it finds.

A quality score is useful when someone can explain what would make it change. If an important condition disappeared from a translation, the measurement should detect that loss. If the translator expressed the same thought more naturally, it should recognize an acceptable difference. Designing a measurement means making those distinctions explicit enough to test.

This guide takes a requirement through definition, calibration, and reporting. It applies whether translations come from a supplier, an internal pipeline, or Lingo.dev. The result is a measurement another person can run and interpret under the same conditions, rather than a number whose meaning depends on its author being present.

Define the change in meaning that matters #

Start with the content's intended effect. An instruction needs to lead to the right action. A claim needs to preserve its strength and qualifications. A product term needs to identify the same concept in another language. “Accurate translation” names the ambition but leaves these distinctions implicit.

Consider certainty and conditions. In may become available after approval, the possibility, the approval requirement, and their relationship all carry meaning. Rendering it as an unconditional promise changes the claim even if the translation retains the words for availability and approval.

A criterion for that distinction could read:

Example criterion: preserve claims and conditions

Compare the source claim with the target claim. Check whether the target preserves:

  • The degree of certainty or obligation.
  • Each condition that limits when the claim applies.
  • The relationship between the condition and the claim.

Allow different wording and sentence structure when they preserve these meanings. For a suspected error, identify the source and target passages and explain which commitment or condition changed. When the available text cannot support the comparison, explain what is missing.

The criterion gives the evaluator something observable to examine. It also gives a reviewer a way to disagree. A vague request to “check accuracy” leaves both the evaluator and the reviewer to supply their own standard.

Choose a scope that fits the next decision. Finding statements that need closer review calls for reliable detection and understandable findings. Comparing providers also needs a common sample and a way to summarize the findings. A decision about publishing a particular item needs the requirements that apply to that item. The same check can contribute to each decision without making all three decisions itself.

Select the text the question can answer #

Define the unit being measured. For the criterion above, use one source claim and its corresponding translated claim, with enough surrounding text to interpret the condition. Keep the content ID, source revision, target revision, and locale together. A correct translation of an older source can appear wrong when compared with the new source.

Select content that contains the distinction being examined. A navigation label with no claim is outside this criterion. Counting it as a pass would improve the rate without showing anything about preserved certainty or conditions.

Missing evidence is a different problem. If a selected claim has no paired source, it still belongs to the work being assessed, but the comparison cannot yet be made. Keep that gap visible. Where it is unclear whether the criterion applies, record the uncertainty rather than quietly dropping the item.

Supply context that can change the answer. A term definition or product rule may resolve an ambiguity; unrelated content may only distract. A target-only spelling check needs less evidence than a source-to-target meaning comparison. The question determines what to collect.

Give each answer a clear meaning #

For the claim-preservation example, three answers are enough:

AnswerMeaning
PreservedThe target retains the source's certainty, obligations, and relevant conditions, allowing equivalent phrasing.
ChangedAn identifiable part of that meaning has changed or disappeared. The finding names the affected passages.
IndeterminateThe evaluator completed the comparison but could not support either conclusion from the evidence. Its explanation identifies the uncertainty.

These are methodological labels. Map them to the actual answer format of the tool being used; they are not an API schema. For example, a Radar custom boolean measurement can return true, false, or a null answer with reasoning. Inspect that reasoning because null can represent more than one cause.

Use a numeric scale when intermediate levels express a useful difference. Define those levels in terms a reviewer can recognize. A value of 80 must mean something more precise than “fairly good.” If severity matters, define it through the consequence of the error, and keep the severe findings visible when combining results. A high average can coexist with an unacceptable change in a critical passage.

A failed request has no completed linguistic answer. Keep it separate from an evaluator that completed and returned an indeterminate answer. That distinction determines whether the next action is to recover execution or resolve the ambiguity in the evidence.

Test the measurement on cases already understood #

Prepare a calibration set from content the team is authorized to evaluate. Include clear errors, good translations, natural paraphrases, and cases whose answers depend on context. Include content outside the criterion's scope to test selection as well as judgment.

Have qualified reviewers establish the expected answers and their reasons. Disagreements are informative. If the source admits two interpretations, settle the intended meaning or preserve the uncertainty. If the standard is clear but the evaluator rejects an acceptable paraphrase, the measurement needs improvement.

Examine two kinds of mistake separately:

  • False alarms are acceptable translations flagged as errors. They consume review attention and can encourage unnecessary edits.
  • Missed errors are real problems the measurement fails to identify. Their consequences depend on the kind of meaning lost, not only how frequently they occur.

Test the revised measurement on examples that were not used to write its instructions. When an LLM supplies the assessment, retain the model, prompt, configuration, and observed variation across repeated runs. Agreement with its own earlier answers does not establish agreement with the team's standard.

Calibration should leave a concrete account of where the check works and where it needs more evidence. A method tested on one content type and language pair is a useful beginning, not a finding about every translation the company produces.

Report the problem and the work examined #

Once the check is useful on known cases, apply it to a defined collection or sample. Record how that sample was chosen. A random sample can support a different inference from a sample selected because another check already considered the translations risky. The latter may help find problems without estimating their frequency across all content.

Consider this illustrative result for 100 selected, eligible translation pairs:

ResultPairs
Meaning preserved72
Meaning changed8
Completed but indeterminate10
No completed evaluation10

Evaluations completed for 90% of the selected pairs. They produced determinate answers for 80%. Among those 80 answers, 8 identified a change, giving a 10% detected-problem rate among determinate answers. The remaining 20 pairs do not establish either preserved or changed meaning.

All three quantities matter. The 72 preserved meanings out of 80 determinate answers give a 90% pass rate among determinate answers. Reporting only “90% passed” would conceal the 10 completed but indeterminate answers and the 10 pairs without a completed evaluation. The 10% detected-problem rate also remains a result of this measurement on this selection; interpreting it as a wider error rate requires a suitable sample and evidence about false alarms and missed errors.

Break the results down by the languages and content types that could change the decision. Show counts as well as rates. A language with little determinate coverage needs investigation before it can disappear into a reassuring overall number. Preserve examples of the recurring problems so the report explains what the team would actually improve.

Keep comparisons meaningful after a change #

Save the criterion and its configuration with the evidence used for the comparison. If the prompt, model, eligibility rule, or source requirement changes, test the new definition before treating its results as continuous with the old ones.

To compare two measurements, evaluate the same preserved text under both definitions. To compare two translation approaches, keep the criteria and relevant conditions consistent. Otherwise a change in the score may describe a new question or easier content rather than improved translations.

Preserve a comparison before rerunning history

Radar's historical evaluation replaces earlier answers for the selected measurement and interval. Save the earlier definitions and results needed for a comparison before using it. A rerun does not retain both sides of the experiment.

The measurement is ready for use when another person can select the intended text, supply its evidence, explain each answer, and reproduce the reported calculation. Use event measurements or local CLI scoring to implement the check. For a continuing external workflow, Measure translations from your pipeline covers collection and recovery; Validate quality findings develops the next task of deciding whether a reported problem is real.