Built-in measurements
Understand built-in quality and edit checks through their inputs, algorithms, answer scales, and missing-evidence behavior.
Built-in measurements answer specific questions about a translation or edit. Edit rate asks whether two texts differ. Glossary adherence asks whether relevant terms were applied. Their values become useful together when each retains its own meaning; they do not form a universal quality scale.
Turn these checks on in Measurements; glossary and rule adherence can also be turned on per engine from its Reviewers tab, as described in Review Lingo.dev translations. This reference explains what each one computes and which evidence can support its answer.
MQM #
MQM asks a model to list the translation's errors and nothing else, never a score. For each error it returns the exact words that are wrong, one of the seven MQM Core dimensions (accuracy, terminology, linguistic conventions, style, locale conventions, audience appropriateness, design and markup), a severity (neutral, minor, major or critical) and one line on what is wrong. When the event names a localization engine, the model also sees that engine's current guidance: its brand voice for the language, its rules, and the glossary terms the source uses, together with the request's context.
MQM also judges translation changes: every target creation and update, whoever made it. In Zendesk and GitHub, a version Lingo.dev wrote, decided as for edit rate below, is judged against the brand voice, rules and glossary terms its translation was made with, and its answer names the job that made it. Any other version, such as a person's edit, a revert, another tool's translation or a Crowdin change, is judged without any engine's guidance. The answer's isLingoRecord field tells the two apart: false for those other versions and true for every other answer, completed localizations included, so Explore can compare Lingo.dev's translations with the rest. Translation changes name no engine, so an MQM condition limited to engines answers them N/A.
The score is arithmetic on that list. Errors whose words do not appear in the translation are dropped. Each remaining error weighs neutral 0, minor 1, major 5 or critical 25, and the total is normalized per 100 source words:
max(0, 100 − 100 × summed error weights / source word count)100 means no errors. The same formula gives a score per dimension. Source words are counted in the text values, not their keys; scripts written without spaces, such as Chinese, Japanese, Korean and Thai, count one word per character. A source with no words is answered N/A without asking the model.
MQM runs on completed localization events and on target creations and updates. Glossary adherence and rule adherence are configured for completed localization events. Edit rate, edit distance and edit patterns run on target creations and updates. Built-in allowed types are platform-defined; inspect the measurement's current settings when enabling it.
Edit rate #
Edit rate compares a target change with the string's previous version: the latest earlier target creation or update for the same string and locale or, when none is recorded, the text before the change that the event carries. It trims leading and trailing whitespace and replaces each internal run of whitespace with one space before comparing. A string without an earlier version compares as unchanged.
Where Lingo.dev writes the translations itself, the three edit measures answer for each version Lingo.dev wrote and for the change a person makes to it. Lingo.dev records each translation it hands over: what its sync writes to Zendesk or commits to GitHub, and what the API, CLI and CI return. A version is Lingo.dev's when it is Lingo.dev's own write, known by its commit or by the time Zendesk stamped on it, or when it holds the latest translation Lingo.dev handed over for the string before that version, such as one the CLI produced. Each translation counts once, on the first version that holds it: a version that brings it back after a person's change is that person's revert.
There, a version Lingo.dev wrote always answers unchanged: edit rate false and edit distance 0, while edit patterns answers not applicable. A person's version that directly follows one Lingo.dev wrote is compared with it: edit rate true, the distance of the change, and its edit patterns. If the person kept Lingo.dev's text, apart from whitespace, the change answers not applicable (“The person kept Lingo.dev's text”); so does a change to a version Lingo.dev did not write, or to no recorded version (“Lingo.dev did not write the version before this one”), including a person's further edits and their revert to Lingo.dev's text. Each false therefore stands for one translation Lingo.dev wrote and each true for one that a person changed: edit rate is the true answers over the false answers, so 5 changed of 100 translations Lingo.dev wrote makes 5%. Average edit distance per translation Lingo.dev wrote is likewise the sum of the changes' distances over the number of Lingo.dev's versions.
In other sources, such as Crowdin, Lingo.dev writes no translations, so a change is compared with the previous version whoever made either.
| Answer | Meaning |
|---|---|
true | The normalized texts differ; in Zendesk and GitHub, a person changed Lingo.dev's version. |
false | The normalized texts match, or the string has no earlier version outside Zendesk and GitHub; in Zendesk and GitHub, Lingo.dev wrote this version. |
| N/A | In Zendesk and GitHub, the person kept Lingo.dev's text, or Lingo.dev did not write the version before this one. |
For example, Save changes and Save changes compare as unchanged. Save changes and Apply changes compare as edited. Neither result decides which wording is better.
If the draft exists but the target field is absent, the implementation compares the draft with an empty target. Check the event's evidence before treating that as a completed human edit. Across a collection, report edited comparisons over usable draft-target comparisons and keep missing comparisons visible.
Edit distance #
Edit distance computes Levenshtein distance on the original, unnormalized texts: the number of insertions, deletions, and substitutions needed to change one into the other. It divides that distance by the longer text's length and multiplies by 100.
Zero means identical text. A higher value means more textual change. Two empty strings produce zero, and so does a string without an earlier version. An absent target is treated as empty, as in edit rate.
The implementation compares UTF-16 code units, not words or user-perceived characters. It skips the start and end the two texts share; when the differing parts still make more than about 3.2 billion character pairs, such as two unrelated texts of 57,000 characters each, the answer is not applicable, with reason “The texts are too long to compare”. Whitespace can contribute to edit distance even when edit rate calls the pair unchanged. A one-unit change also contributes more to the percentage of a short string than of a long one.
Use the two edit measures together to distinguish frequent small changes from occasional extensive rewrites. Outside Zendesk and GitHub, verify the pairing first: the previous version can be an earlier edit rather than the delivered translation. Neither measure establishes errors corrected, time spent, or quality gained.
When aggregating distance, name the calculation. Averaging per-pair percentages gives each pair equal weight; dividing total raw distance by total comparison length gives longer texts more weight. An individual event answer does not choose between those formulas.
Glossary adherence #
Glossary adherence compares source and target against terms attached to the localization engine when evaluation runs. It requires source text, target text, and a localization engine identity. It selects attached terms whose target locale matches and whose source text appears in the source, using a case-insensitive substring check.
The judge evaluates each supplied term. Its instructions allow natural grammatical inflection of custom translations, require non-translatable terms to remain unchanged, allow a standard rendering when a context hint clearly does not apply, and accept one of several conflicting entries for the same term. A high score therefore does not establish that conflicting glossary definitions are good policy.
The score is:
returned findings marked applied / all returned term findings × 100This counts returned findings, rather than every occurrence of a term or every entry in the organization's glossary. Missed terms contribute to the explanation. If the judge returns no findings, the completed answer has no determinate score.
Missing required fields, no matching localization engine or attached terms, and no terms selected from the source can produce N/A before a judge runs. Read the reason. A genuinely irrelevant term needs no judgment; a message that should have had terminology evidence still has an evidence gap.
This measurement reads current guidance, on live events and in a backfill alike. It does not reconstruct the glossary used during generation. Requests shows the source batch and model attempts, without a full snapshot of the prompt or glossary. Save the submitted context and relevant term definitions before comparing historical event evaluations, then use Glossaries for accepted terminology changes.
Rule adherence #
Rule adherence uses source text, target text, a localization engine identity, and current attached rules applicable to the target locale. The judge returns one finding per supplied rule: followed, not followed, or null when the rule does not apply.
The score is:
returned findings marked followed / returned non-null rule findings × 100Null rule findings are excluded. If all returned findings are null, the judge can complete without an overall score. Missing source, target, or localization engine evidence can instead prevent the judge from running. Both paths can appear as N/A, so inspect the reason and available execution evidence.
Read the failed rule and the words it describes before changing the translation. An ambiguous or contradictory rule may need clarification in Rules. As with glossary adherence, historical event evaluation uses current attached guidance, rather than a generation-time snapshot.
Edit patterns #
Edit-pattern analysis asks a model to classify changes between the draft and target. It can assign several categories to one comparison:
| Label | Change described |
|---|---|
register | Formality or politeness. |
word_choice | Different wording with the same meaning. |
terminology | A product or domain term. |
reordering | Word or clause order. |
grammar | Agreement, case, tense, or other grammar. |
punctuation | Punctuation or spacing. |
addition | Meaning added that the draft omitted. |
omission | Meaning removed that the draft added. |
mistranslation | A changed interpretation of meaning. |
formatting | Placeholders, markup, capitalization, or numbers. |
The classifier skips strings without an earlier version and texts that match after whitespace normalization. Its labels describe suspected kinds of change, rather than independently confirmed defects or time savings.
The current result list and detail read paths filter to boolean and numeric answers. Although the detail contract has an answer field for structured data, those paths do not currently return the stored JSON edit-pattern rows. Use the accessible edit comparisons and inspected text rather than expecting a categorized export from those operations.
Usage and interpretation #
Edit rate and edit distance compute answers without LLM spend. Successful measurement activity has separate usage accounting, so zero model cost is not a promise of free measurement. Costs explains the accounting boundary; Results explains null answers and coverage.