Which metric evaluates generative text summarization accuracy in Amazon Bedrock?

AIF-C01 By islever Editorial Last updated Aug 3, 2026

Short answer

C BERTScore

Question

A company has developed a generative text summarization model by using Amazon Bedrock. The company will use Amazon Bedrock automatic model evaluation capabilities. Which metric should the company use to evaluate the accuracy of the model?

Area Under the ROC Curve (AUC) score
F1 score
BERTScore Correct Answer
Real world knowledge (RWK) score

Key Takeaways

This question tests your ability to match evaluation metrics to generative AI tasks; BERTScore is purpose-built for text generation quality (summarization, translation), while AUC, F1, and RWK serve different purposes.

When evaluating a generative text summarization model with Amazon Bedrock's automatic model evaluation, BERTScore is the correct metric because it measures semantic similarity between generated and reference text using contextual embeddings.

Common mistake: Candidates often pick F1 score because it is a well-known accuracy metric in traditional ML classification tasks, but F1 measures token-level overlap and does not capture semantic meaning in generated text.

Expert Analysis

Understanding Amazon Bedrock Automatic Model Evaluation Metrics

When using Amazon Bedrock's automatic model evaluation capabilities, it is critical to choose the correct metric for the type of model and task you are evaluating. The question specifically asks about evaluating the accuracy of a generative text summarization model.

Why BERTScore (Option C) is Correct

BERTScore is a metric specifically designed for evaluating text generation tasks such as summarization, translation, and paraphrasing. It works by leveraging contextual embeddings from pre-trained models like BERT to measure the semantic similarity between the generated text and the reference (ground truth) text. Unlike traditional metrics that rely on exact token matching, BERTScore captures deeper semantic relationships, making it ideal for assessing whether a summary accurately conveys the key points of the original document.

As community members noted, BERTScore aligns perfectly with the goal of text summarization — capturing meaning rather than matching words exactly.

Why the Other Options Are Incorrect

Option A: Area Under the ROC Curve (AUC) score — AUC is used for evaluating binary classification models, measuring the model's ability to distinguish between positive and negative classes. It has no relevance to text generation or summarization tasks.
Option B: F1 score — While F1 score is a popular metric in NLP, it is primarily used for classification tasks or token-level extraction tasks (like named entity recognition). It measures the harmonic mean of precision and recall based on exact token overlap, which fails to capture semantic similarity in free-form generated text.
Option D: Real World Knowledge (RWK) score — RWK evaluates how well a model retrieves and uses factual knowledge from its training data. While useful for open-domain question answering, it is not the primary metric for summarization accuracy, which focuses on how faithfully the model condenses a given input text.

Key Takeaway

Amazon Bedrock supports multiple evaluation metrics, and the right choice depends on the task:

Text generation / summarization → BERTScore
Classification → AUC, F1
Knowledge-based QA → RWK

Official Reference

https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation.html

This link opens the official vendor documentation in a new tab.

Exam Strategy

When you see a question about evaluating generative AI models on AWS, immediately identify the task type (classification vs. text generation vs. knowledge retrieval) and match it to the appropriate metric. BERTScore is the go-to answer for any text generation or summarization evaluation scenario in Amazon Bedrock.

Community Votes

100%

100% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 4 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 4

ap6491 👍 1 Answered C

BERTScore is a metric specifically designed to evaluate text generation tasks, such as summarization. It measures the semantic similarity between the generated text and the reference text by leveraging contextual embeddings from pre-trained models like BERT. BERTScore captures deeper semantic relationships, making it ideal for evaluating the accuracy and meaningfulness of summaries.
Jessiii 👍 1 Answered C

BERTScore: This metric leverages the capabilities of a pre-trained BERT model to assess the semantic similarity between the generated summaries and the reference text, providing a more accurate evaluation of the model's ability to capture the key points of the original text, which is crucial for text summarization.
aws_Tamilan 👍 1 Answered C

BERTScore is the most appropriate metric for evaluating the accuracy of a generative text summarization model because it compares semantic similarity in a manner that aligns well with the goal of text summarization.
may2021_r 👍 1 Answered C

The correct answer is C. BERTScore is specifically designed for evaluating text generation quality.

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Practice All AIF-C01 Questions

Access 100 questions with complete answers and detailed explanations.

View Full AIF-C01 Practice Test →

← Back to AIF-C01 Study Guide