Evaluating RAG Answer Quality with Measurable Techniques

AI-300 By islever Editorial Last updated

Configure evaluation and validation for generative AI applications and agents

Short answer

B Reviewing user feedback comments is subjective and does not provide the measurable evaluation required to validate RAG answer groundedness.

Question

Note: This question is part of a series of questions that present the same scenario. Each question in the series contains a unique solution that might meet the stated goals. Some question sets might have more than one correct solution, while others might not have a correct solution.
After you answer a question in this section, you will NOT be able to return to it. As a result, these questions will not appear on the review screen.
You manage a Retrieval-Augmented Generation (RAG) application built on Microsoft Foundry.
The application retrieves documents from an indexed knowledge base and generates answers for internal users.
Recent feedback indicates that answers are fluent but sometimes include information that is not supported by the documents that were retrieved.
You need to evaluate whether a proposed change improves RAG answer quality by using supported and measurable techniques.
Solution: Review user feedback comments collected after deployment to determine whether answers appear more accurate.
Does the solution meet the goal?
  1. Yes
  2. No Correct Answer

Key Takeaways

The exam tests your ability to identify measurable RAG evaluation techniques; the common trap is confusing qualitative user sentiment with quantitative accuracy metrics like Groundedness.

This question tests the distinction between subjective user feedback and objective, measurable evaluation metrics for Retrieval-Augmented Generation (RAG) applications. It establishes that relying on unstructured comments is insufficient for validating improvements in groundedness.

Common mistake: Learners often select 'Yes' because they assume user feedback is a valid form of quality assurance, failing to recognize that it lacks the measurability required by the prompt's specific constraints.

Expert Analysis

Why the Answer Is Correct

The correct answer is B because the solution relies on reviewing user feedback comments, which are inherently subjective and unstructured. The question explicitly requires 'supported and measurable techniques' to evaluate if answers are fluent but unsupported (a grounding issue). User feedback cannot reliably quantify whether retrieved documents actually support the generated text.

Why the Other Options Are Wrong

Option A is incorrect because it validates a non-measurable approach. In Microsoft Foundry and Azure AI Search contexts, evaluating RAG quality requires structured metrics such as Groundedness (are claims supported by sources?), Relevance (does it answer the query?), and Faithfulness. Without these quantifiable scores, one cannot objectively determine if a change improved quality.

Community Comment Notes

Community consensus strongly supports Option B, noting that subjective feedback does not provide standardized comparison. As DAgoGi noted, "Reviewing user feedback comments is subjective and does not provide a measurable evaluation," highlighting the need for metrics like Groundedness and Relevance. Kachien also emphasized that this symptom is specifically a "groundedness problem" requiring technical validation rather than anecdotal evidence.

Exam Strategy

Always look for keywords like 'measurable', 'quantifiable', or 'objective' when evaluating GenAI solutions. If the proposed method relies on human opinion or logs without structured scoring, it is likely incorrect for technical validation questions.

Frequently Asked Questions

Why is user feedback not considered measurable?

User feedback is unstructured text or ratings that vary by individual interpretation. Measurable evaluation requires consistent, algorithmic scoring of attributes like Groundedness against a ground truth dataset.

What metrics should be used instead of user feedback?

Use automated evaluation metrics such as Groundedness (support from source), Relevance (alignment with query), and Faithfulness to quantitatively assess RAG performance changes.

More AI-300 FAQ →

Discussion 2 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 2

  1. DAgoGi 👍 1 Answered B
    According to GPT 5.6 the correct answer: B. No. Reviewing user feedback comments is subjective and does not provide a measurable evaluation of RAG answer quality. To determine whether the change improves answer quality, use measurable evaluation metrics such as: Groundedness: Are the answers supported by the retrieved documents? Relevance: Does the answer address the user's question? Retrieval quality: Are the retrieved documents relevant to the query? These metrics allow you to compare performance before and after the proposed change.
  2. Kachien 👍 1 Answered B
    Reviewing user feedback comments is subjective and unstructured. It does not provide a standardized, measurable comparison of whether the proposed change reduces unsupported information. The symptom described is a groundedness problem.

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Submitted for moderation before publishing. Keep it helpful and respectful.

Azure ML Asset Registration for Reusability

S-grade

Assessing and Improving RAG Retrieval Quality Sequence

S-grade

Implementing RAG Flow with Microsoft Foundry and Azure OpenAI

S-grade

Accessing Folder Data Assets in Azure ML

A-grade

← Back to AI-300 Study Guide