Measure translations from your pipeline
Add repeatable measurement to an existing translation workflow while keeping revisions, answers, and incomplete work identifiable.
An existing translation workflow already knows where its content comes from and where accepted text belongs. Add measurement at a point where the source and target versions can be preserved, then return each answer to that record. The team can investigate quality without changing which system produces or maintains translations.
This guide follows one recurring measurement from collection through recovery and reporting. Begin with a defined criterion, such as preserving conditions in product claims. Design a translation-quality measurement explains how to write and calibrate that criterion.
Connect the check to a real intake #
Choose the execution path before designing automation around it:
| Existing work | Measurement path |
|---|---|
| Crowdin suggestions and translations | Connect Crowdin, enable an event measurement, and inspect collected activity and results in Radar. |
| Local translation files | Run lingo score with a scorer and model configuration; retain its local result file. |
| Lingo.dev translation requests | Run a measurement on their completed localizations and read the answers in Radar. |
Radar's collected-event source determines which payload arrives. It does not provide a generic endpoint for arbitrary translation or score uploads. For a self-built pipeline without a supported intake, preserve the revision record below and establish the transport before automating submission. A custom criterion by itself does not create that transport.
Collect translations explains these boundaries. The measurement method remains useful wherever accepted text is maintained; local scoring and event history simply produce different records to reconcile.
Keep the source and target versions together #
Choose a content identity that survives edits, then record the particular source and target revisions evaluated. A message ID alone cannot distinguish an answer about yesterday's text from an answer about today's correction.
An illustrative customer-owned record might contain:
{
"contentId": "checkout.delivery",
"sourceRevision": "source-17",
"targetRevision": "fr-12",
"targetLocale": "fr",
"criterionVersion": "delivery-promise-3",
"sourceText": "Your order will arrive within five business days.",
"targetTextLocation": "accepted catalog at revision fr-12"
}This is a tracking example, not a Lingo.dev request schema. Store actual target text or an immutable reference available to the evaluator. A link to “latest” can change while evaluation is running. Keep occurrence time, producer when known, and the location of accepted text with the record.
Add evidence according to the question. Meaning preservation needs paired source and target; edit activity needs the actual draft; terminology can need a definition and usage context. A spelling check may need only target text. Missing evidence for an eligible message remains a gap in the assessment, rather than a reason to count the message as passing or irrelevant.
Complete one collection-to-answer cycle #
For Crowdin, follow the connection instructions, then configure the event measurement and its trigger and target locales. For Lingo.dev translations, verify the measurement's engines, source and target languages, and sample. Save those settings before producing the test activity or request.
Find the resulting event or request and compare its text with the preserved record. Then inspect the answer with the criterion. A received event, selected evaluation, and completed answer are separate milestones. A new selection setting does not retroactively evaluate old events; use historical evaluation when that is the intended task.
Retain a simple reconciliation record in the workflow: selected revision, received event or request identity, measurement identity, completed answer, and any unresolved gap. Events and Results explain the fields actually available. Local scoring instead needs its source files, configuration, and output file preserved together.
Handle repeats and corrections differently #
A retry delivers the same revision again. A correction creates a new target revision, even when the content ID stays the same. Keep that distinction in your record before applying the source's own deduplication behavior.
For example, a corrected deadline should receive a new target revision and a new comparison with the applicable source. Retain the earlier text and answer if you need to show what improved. If the source promise also changed, preserve its new revision; comparing the new target with the old source can create an apparent defect.
Provider deliveries do not share one unlimited idempotency guarantee. Repeated Crowdin review actions can have distinct event identities, while other activity uses its own occurrence information. Reconcile available receipts and results before repeating work whose outcome is uncertain.
Recover from the last established milestone #
| What you can establish | Next investigation |
|---|---|
| The source contains the revision, but no event is found | Check the connection, supported activity type, time window, and collection evidence. |
| The event exists, but no answer is found | Check enabled measurement conditions, locales, required payload, and asynchronous processing. |
| The judge completed with an indeterminate answer | Read its explanation and supply missing evidence or resolve the ambiguity. |
| The criterion genuinely does not apply | Retain that decision and exclude the item from this criterion's quality denominator. |
| Execution failed | Identify the failure before retrying the affected work through its supported path. |
Absence alone does not distinguish waiting, failure, dropped input, or unrequested work. Keep that uncertainty until evidence resolves it. Repeated generation is not a reliable repair for a missing asynchronous evaluation.
If recovery uses historical evaluation, keep Only events not measured yet on to fill gaps without touching existing answers. Measuring the period again replaces earlier answers for that measurement throughout the selected interval, so preserve them first. Confirm the expected events and returned answers rather than treating the start response as completion.
Report the answers and the unanswered work #
Consider an illustrative batch of 50 eligible revisions. Evaluations completed for 45 and produced determinate answers for 40. Of those, 36 preserved the requirement and 4 failed it.
Completion coverage is 45/50, or 90%. Determinate coverage is 40/50, or 80%. The pass rate among determinate answers is 36/40, or 90%. The five completed-but-indeterminate answers and five revisions without completed evaluations remain separate.
Show the same counts by language or content class where missing work is uneven. Keep unknown applicability visible until it is resolved. A high pass rate for answered records cannot describe the missing ones.
Keep the criterion with the result #
Preserve its prompt, evidence requirements, answer definitions, and evaluator configuration with the content revisions. Test changed definitions on known examples before combining their answers into a trend. An altered criterion can change scores without changing translations.
When a finding is confirmed, correct the text in its owning system and evaluate the new revision. Validate quality findings helps establish that diagnosis. Measurement supplies evidence for the decision; writeback and publication remain explicit actions in the existing workflow.