AI Reviewers

AI reviews are automated checks that examine the translations your localization engines produce. In the dashboard they are called measurements, and they live under Radar → Measurements. The two names mean the same thing. A measurement answers one question about a translation - pass or fail, or a score from 0 to 100 - and explains its answer. Reviews run asynchronously after the translation completes: they never block it, change it, or decide whether it ships.

How it works#

Every completed localization is recorded in Radar as a localization.completed event, carrying the source and target text, both locales, the engine, and the request's context. Each measurement that runs on that event type, and whose conditions select the event, queues an evaluation. The answer is stored with the event it examined.

There is no separate reviewer resource for engine translations. The same measurements can also examine translation changes collected from connected sources, such as a GitHub repository or a Zendesk Help Center - that is where the edit measurements below get their evidence.

Radar → Measurements lists Your measurements and Built in, with the event types each runs on, its value type, its languages, how many events it has measured, and whether it is on.

Built-in measurements#

Lingo.dev defines what each built-in computes and which event types it can run on. Your organization decides whether it runs and which events it selects.

To turn one on, open it in Radar → Measurements, choose the languages, source languages, engines, and sample it should examine, turn on Enabled, and select Save changes. Built-ins cannot be edited or deleted. Glossary adherence and rule adherence can also be switched on per engine - see Turn on reviews for an engine.

Glossary adherence #

Checks whether the translation applied the glossary terms the engine applies. It selects the terms whose target locale matches and whose source text appears in the source, and the judge returns a finding for each one. Custom translations (e.g., "Deploy" → "Bereitstellen") may inflect naturally, non-translatable terms (e.g., "OAuth") must stay unchanged, and when entries for the same term conflict, following one of them counts.

The score is the share of returned findings marked applied. When no glossary term applies to the source, the answer is N/A.

Rule adherence #

Evaluates each rule the engine applies for the target locale independently: followed, not followed, or not applicable. If the engine applies three rules, the judge returns three findings, each with its own reasoning.

The score is the share of followed rules among the rules that applied. A rule that does not apply to the content - for example, a formal-address rule on a translation that contains only a product name - is left out. If no rule applies, there is no score.

Both checks read the engine's glossary terms and rules as they are when the check runs, not a snapshot of what the engine used when it generated the translation.

MQM#

MQM asks a model to list the translation's errors, never to score it. Each error names the exact words that are wrong, one of the seven MQM Core dimensions - accuracy, terminology, linguistic conventions, style, locale conventions, audience appropriateness, design and markup - a severity, and one line on what is wrong. When the event names an engine, the model also sees the engine's brand voice for the language, its rules, and the glossary terms the source uses.

The score is arithmetic on that list. Errors whose words do not appear in the translation are dropped. Each remaining error weighs neutral 0, minor 1, major 5, or critical 25, and the total is normalized per 100 source words:

max(0, 100 − 100 × summed error weights / source word count)

100 means no errors, and the same formula gives a score per dimension. Scripts written without spaces, such as Chinese, Japanese, Korean, and Thai, count one word per character. A source with no words answers N/A.

Edit rate, edit distance, and edit patterns#

These three compare a translation change with the string's previous version. Where Lingo.dev writes the translations itself - a Zendesk Help Center sync, or a repository where the GitHub App commits translations or the CLI pushes them - they answer once for each version Lingo.dev wrote, and once for the change a person makes to it:

• A version Lingo.dev wrote answers unchanged: edit rate false and edit distance 0.
• A person's version that directly follows one Lingo.dev wrote is compared with it: edit rate true, the distance of the change, and its edit patterns.
• A person's further edits, a revert to Lingo.dev's text, and a change that keeps Lingo.dev's text apart from whitespace answer N/A.

Each false therefore stands for one translation Lingo.dev wrote, and each true for one that a person changed. 5 changed out of 100 translations Lingo.dev wrote is an edit rate of 5%.

Edit rate trims leading and trailing whitespace and collapses internal whitespace before comparing, so Save changes and Save changes count as unchanged. Edit distance is the Levenshtein distance between the original texts - insertions, deletions, and substitutions - divided by the longer text's length and multiplied by 100. Texts too long to compare answer N/A. Read the two together to tell frequent small fixes from occasional rewrites. Edit patterns classifies a change into categories such as register, word choice, terminology, reordering, grammar, punctuation, addition, omission, mistranslation, and formatting.

None of the three judges which wording is better, or measures time spent.

Turn on reviews for an engine #

The engine's Reviewers tab has two sections:

Automatic review has a switch for Glossary adherence and one for Rule adherence. Turning a switch on adds the engine to that built-in measurement and turns the measurement on; turning it off removes the engine. The measurement's language and sample settings stay as they are. When a built-in runs on every engine, each engine shows its switch on - turning one engine off stores the list of the other current engines, so an engine created afterwards is not reviewed until it is added. Where your plan meters review, the section shows Billed per run with the rate.

Suggestions holds the switches for Engine Suggestions, which act on low custom-measurement answers rather than produce them.

MQM and custom measurements are not on this tab. Turn MQM on in Radar, and select the engines for a custom measurement in its condition.

Custom measurements #

A custom measurement makes one of your own quality requirements explicit enough to check: preserving a qualification, following a voice requirement, or keeping two product concepts distinct. It belongs to the organization.

Pass or fail answers are aggregated as pass rates - 75% means 3 of 4 answered translations passed. 0 to 100 answers are aggregated as averages. A percentage is a value on the scale your prompt defines, not a probability that the translation is correct.

Choose which translations are reviewed#

Each condition - and each built-in's settings - narrows the events of its type:

Select Save changes to apply them. Locale selection follows the locale hierarchy rather than exact text: fr also selects fr-CA, and fr-CA does not select fr-FR. A request recorded without an engine does not pass an engine-limited condition.

Sampling is decided per event and measurement, so the same event falls on the same side of a given sample when it is evaluated again. Over a small batch, the sampled share can differ from the configured one.

An event of a selected type that fails a selector still receives an answer: N/A, with reasoning that begins "Condition not met" and names the selector. It is not billed. Keep these answers out of a quality denominator, but account for them when you explain how much of your translation volume was reviewed.

Enabled turns all of a measurement's conditions on or off together. Turning it off stops future reviews; work already queued may still complete.

Writing instructions #

The prompt is the core of a custom measurement. It tells the judge exactly what to check. Write it as a specific, testable criterion.

Good instructions#

Pass or fail:

Check whether all HTML tags in the source text are preserved
exactly in the translation. Tags must not be added, removed,
modified, or reordered. Pass if all tags are preserved, fail
if any tag is missing or altered.

0 to 100:

Rate the fluency of the translation on a scale of 0-100.
100 means a native speaker would find it completely natural.
0 means it reads like machine output. Deduct points for
awkward phrasing, unnatural word order, or overly literal
constructions.

What makes a good instruction#

• Specific criteria - define exactly what pass and fail mean, or what 0 and 100 represent
• Observable outcomes - the judge should be able to decide by reading the text, not by guessing intent
• One concern per measurement - split multi-dimensional quality checks into separate measurements
• When to abstain - with Allow not applicable on, say when the criterion does not apply, so an irrelevant translation is not confused with one that is missing evidence

Test a new measurement on translations you can inspect - known errors and acceptable paraphrases alike - before relying on its answers across many translations.

N/A answers#

With Allow not applicable on, the judge can answer N/A instead of a verdict or score - for example, a formal-address check on a language without a formal and informal distinction. N/A answers are excluded from averages and pass rates; they don't pull scores down or inflate them. With it off, the judge must give a pass, a fail, or a score. An event that fails a condition is answered N/A without asking the judge, whatever this setting.

Read the answers#

Each answer carries reasoning and, when the evidence supports one, a suggested correction.

• Radar → Events → Measurements lists every answer. Filter by Measurement and Value, then open one for its value, reasoning, suggestion, and the judge's request.
• The engine's Logs tab has a Review column for each translation request. It averages the latest answer of each custom measurement, counting a pass as 100 and a fail as 0 and leaving out N/A. It shows N/A when every answer was N/A and Not reviewed when there is none. Glossary adherence, rule adherence, and MQM are not part of this column.
• Reports chart answers over time on the Radar overview and in Explore.

A failing or low custom-measurement answer on an engine's translation can also trigger Engine Suggestions. The answer itself does not change the translation.

Review translations you already have#

A measurement turned on today examines new events. To apply it to events already collected, open its Backfill tab and choose an interval. Backfill can measure only the events that have no answer yet, or measure the interval again - which replaces the measurement's earlier answers in it. Save the definitions and results you need for a comparison before measuring again.

Review model#

You don't pick the evaluation model. Glossary adherence, rule adherence, MQM, edit patterns, and custom measurements are judged by a model the platform sets, configured separately from the models the engine translates with. Edit rate and edit distance are computed without a model.

Change or remove a measurement#

Edit a custom measurement's prompt, name, value type, or Allow not applicable on its General tab, together with its conditions, and select Save changes. Changing the criterion changes what later answers mean, so keep the earlier definition with any results you will compare against.

Deleting a custom measurement removes its definition and conditions. Its recorded answers remain, shown as Unknown measurement, and no longer count in the engine's Review column. To stop future reviews without losing the definition, turn it off instead.

Managing measurements via MCP#

If you use the Lingo.dev MCP server, your AI coding assistant can list, create, and update measurements, set their conditions, turn them on or off, and read their answers:

"Create a pass-or-fail measurement on completed localizations
that checks whether HTML tags are preserved in translations."

"Add a 0 to 100 fluency measurement for English to German
translations from the docs engine, sampling 50% of them."

Next Steps#