Quality that can be understood
Our responsibility is to make changes in meaning visible and measurable, so localization teams can act on their own standards.
Localization teams deserve to understand the quality of their translations. A native speaker can miss domain-specific nuances. An expensive model can produce slop. Measurement should reveal how a translation changes the meaning: where a precise term became ambiguous, a possibility became a promise, or an important condition disappeared. It should also show how often these changes occur and which languages are affected.
That understanding matters because the team is responsible for content it cannot read in every language. An overall impression of fluency offers little help when the question is whether a reader will understand the right action or believe the right claim. Useful measurement connects the translated words to what the author intended them to do.
Preserve the decisions made in writing #
When authors revise content, they decide what deserves explanation, what can be assumed, and which argument will persuade the reader. A qualification can make a claim accurate. A carefully chosen term can distinguish two product concepts. A restrained sentence can earn confidence without overstating the evidence.
Translation should carry those decisions into another language. Sentence structure may change substantially while the meaning remains precise. Conversely, familiar words and smooth phrasing can conceal a change in the author's claim. Counting matching terms or judging naturalness alone cannot explain that difference.
We treat the team's requirements as the basis for measurement. Those requirements connect language to the work the content must accomplish. Product knowledge explains the intended action; language expertise explains what a reader is likely to understand. Software should help those contributions reach each other, especially when they belong to different people.
Make measurement open to examination #
A quality score should be easy to explain: what was checked, how it was checked, and how the results were combined. Teams should have the tools to define those criteria around what matters in their content.
This sets a demanding standard for our contribution. A finding needs to identify the passage and explain the suspected change. A method needs to distinguish an acceptable paraphrase from a loss of meaning. A report needs to show whether its conclusion describes a few selected examples, a representative sample, or the full measured collection. Without those connections, the team receives a number but inherits the work of discovering what it means.
The measurement itself also deserves scrutiny. A false alarm can divert a qualified reviewer from real problems. A missed condition can matter even when the average score improves. Measurement should let teams test a check against examples they understand, examine disagreements, and distinguish errors it misses from acceptable writing it flags. Those mistakes have different consequences, and combining them into an overall agreement rate can hide the one that matters for the job.
Turn findings into useful decisions #
A quality report should not leave a localization manager guessing what to fix. It should show which passages miss the team's requirements and which problems keep recurring.
Those patterns lead to different kinds of work. A mistranslated sentence needs a correction. A recurring ambiguity may need better product context. An inconsistent term can reveal conflicting glossary entries. A concentration of errors in one language pair can justify a closer comparison of providers or models. Treating all four as a low score conceals the choice the team actually faces.
Our contribution is to make the evidence useful for those choices. A provider comparison needs the same content and criteria on both sides, with enough coverage to examine the languages that matter. A report should distinguish a recurring problem from a difference caused by the sample, and preserve the serious findings that an average can conceal. The team can then weigh the observed differences against its content, budget, and responsibilities, and explain why a particular tradeoff is acceptable.
Keep understanding as the work changes #
A new model, a revised glossary, or a different mix of content can change the results. Useful measurement must make those changes interpretable. Comparing two averages is insufficient if one includes difficult new languages and the other covers familiar material, or if the definition of an error changed between them. Rechecking preserved examples under both definitions helps separate a changed measurement from a changed translation. Reporting unanswered cases by language shows where the comparison still lacks evidence.
We believe localization infrastructure should preserve the relationship between a requirement, the text examined, and the evidence returned. That relationship lets a team learn from repeated work without confusing more activity with better quality. It also makes disagreement productive: people can examine the same passage and criterion instead of debating whether to trust a score.
The measurement-design guide develops this into a practical method. The purpose is a localization team that can explain the quality of its work, locate the problems that deserve attention, and choose what to change with a clear view of the evidence.