Choosing BLEU to Evaluate Machine Translations Against References
Short answer
B Use the BLEU score to automatically score the model's translations against the human reference translations across all languages.
Question
Which evaluation metric will meet these requirements?
Key Takeaways
The question tests knowing which text-generation metric matches which task; the trap is picking ROUGE because the scenario also mentions article summaries, when only translation quality against references is being scored.
This AIF-C01 question tests automatic evaluation of machine translation against human references in Amazon SageMaker AI. The correct metric is the BLEU score (B), which measures n-gram precision versus reference translations.
Common mistake: Choosing ROUGE (A) because the scenario involves article summaries; ROUGE is recall-oriented for summarization and does not properly score translation quality against reference translations.
Expert Analysis
Why BLEU fits The question asks for an automatic metric that scores machine translations against human reference translations across multiple languages. BLEU (Bilingual Evaluation Understudy) is exactly that: it computes modified n-gram precision between the candidate translation and one or more human reference translations, then applies a brevity penalty, producing a 0–1 score that is language-agnostic and standard in MT evaluation.
The attractive wrong answer: ROUGE ROUGE is the trap here, because the same team uses SageMaker to generate article summaries, and ROUGE is the well-known summarization metric (recall-oriented, measuring n-gram recall of generated text against references). Option A therefore satisfies the "article summaries" half of the scenario and is the option test-takers reach for first. The break point is the specific requirement: score the translations against reference translations. ROUGE's recall-oriented design rewards coverage of reference content, which penalises faithful translations that reorder or rephrase; it is not the metric designed for translation quality. Once you notice the question says "score the quality of the model's translations against the references," ROUGE stops being correct and BLEU's precision-based n-gram matching takes over.
Why C and D are wrong Option C (ROC/AUC) measures a classifier's ranking ability across thresholds and requires positive/negative labels, not reference text. Option D (Precision) is a single-classification quantity, not a corpus-level text-generation metric; BLEU in fact generalises precision over n-grams, which is why D is a distractor built from part of BLEU's definition.
Verdict: Option B BLEU is the automatic metric that meets the translation-against-references requirement.
Official Reference
https://docs.aws.amazon.com/sagemaker/latest/dg/text2text.html
This link opens the official vendor documentation in a new tab.
Exam Strategy
When a question names a target output type (translation, summarization, classification) plus a reference, match the metric to that output: BLEU for translation, ROUGE for summarization, ROC-AUC for classifiers. Ignore scenario details that describe other pipeline stages — here the summary step exists only to bait you into ROUGE. Read the final requirement sentence last and let it decide.
Frequently Asked Questions
Why is ROUGE not suitable for scoring machine translations?
What exactly does a BLEU score measure?
Can ROC or Precision evaluate translated text?
Community Votes
100% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.
Discussion 1 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 1
-
NareshGondi 👍 1 Answered BBLEU score is the standard industry metric specifically designed to evaluate the quality of machine-translated text by comparing it against one or more human reference translations. It scores translations based on n-gram precision across multiple languages. ROUGE is primarily used for evaluating text summarization rather than translation, as it focuses heavily on recall ROC and Precision are traditional machine learning evaluation metrics used for classification tasks
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.
Related Analysis
Practice All AIF-C01 Questions
Access 100 questions with complete answers and detailed explanations.
View Full AIF-C01 Practice Test →