Which Evaluator Isolates RAG Retrieval Failures per Agent
Short answer
Drag the document retrieval evaluator to both the support agent and the policy agent, because the failures occur at retrieval, not at response generation.
Question
You have a multi-agent Retrieval-Augmented Generation (RAG) solution that uses a Foundry IQ knowledge base. The solution includes a support agent and a policy agent.
You have an evaluation dataset that contains the following for each user query:
The expected source IDs
Retrieved chunks in rank order
The final agent response
You discover that an embedding model change and a custom analyzer change cause the failed traces shown in the following table.
You need to isolate the failing parts of the RAG pipeline.
Which evaluator should you use for each agent? To answer, drag the appropriate evaluators to the correct agents. Each evaluator may be used once, more than once, or not at all. You may need to drag the split bar between panes or scroll to view content.
NOTE: Each correct selection is worth one point.
Key Takeaways
The question tests whether you can map failure symptoms (expected source IDs and rank order of retrieved chunks) to the retrieval-stage evaluator, and the trap is reflexively choosing groundedness or answer-relevance because those are the most familiar RAG metrics.
When an embedding model change and a custom analyzer change break a Foundry IQ knowledge base RAG pipeline, the failure is in chunk retrieval and ranking, not in answer generation. The correct pairing for this drag-and-drop is the document retrieval evaluator applied to both the support agent and the policy agent.
Common mistake: Learners assign groundedness, answer relevance, fluency, or coherence evaluators because those dominate RAG evaluation discussion, but those scorers measure the generated response against retrieved context and cannot detect that an embedding or analyzer change shifted which chunks were returned.
Expert Analysis
Why the Answer Is Correct
Both failing traces originate before generation: the embedding model and the custom analyzer directly determine which chunks are retrieved and in what rank order, so the discrepancy shows up between the expected source IDs and the retrieved chunks in rank order. Only the document retrieval evaluator compares expected source IDs against the retrieved chunk list, which is exactly the data your evaluation dataset already contains for each user query. Because both the support agent and the policy agent pull from the same Foundry IQ knowledge base, both are exposed to the same retrieval degradation and both need the document retrieval evaluator. This lets you isolate the pipeline stage: if document retrieval scores drop while answer-level scores stay stable, the break is in indexing/retrieval rather than prompt or model behavior.Why Other Candidate Answers Are Wrong
Groundedness measures whether the final agent response is supported by the retrieved context, and a learner correctly pointed out that grounding is not implicated because the agents still produce answers backed by whatever they retrieved. Answer relevance, similarity, completeness, coherence, and fluency all score the generated text itself, so they would not reveal that an analyzer or embedding change altered chunk selection. Tool-call, task-adherence, and safety evaluators evaluate orchestration and policy behavior, which are untouched by a search-index configuration change. Assigning any of these to either agent would mask the actual failure point instead of isolating it.Community Comment Notes
The most useful comment argued that groundedness is not an issue in either case because the agents retrieve information as expected but the information itself is incomplete or irrelevant to what was needed. That same commenter concluded that among the available choices only document retrieval is connected to the problem, since the remaining evaluators address the generative response rather than the knowledge used. This matches the trace evidence in the exhibit: the failures sit in the retrieval stage shared by both agents, so the same evaluator belongs in both slots.Official Reference
https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/evaluate-sdk
This link opens the official vendor documentation in a new tab.
Exam Strategy
Read the failure evidence first: if the dataset compares expected source IDs with rank-ordered retrieved chunks, choose retrieval-stage evaluators; if it compared the response to context, then groundedness would apply. Apply the same reasoning to each agent pane independently, but do not assume the two panes must differ — evaluators may be reused.
Frequently Asked Questions
Why isn't groundedness the right evaluator for these failed traces?
Why do both agents get the same evaluator in this drag-and-drop?
Discussion 1 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 1
-
95624c6f 👍 1Groundedness is not an issue in either of these cases (agent retrieves information as expected, but the information is incomplete or irrelevant). Out of possible answers here, only "document retrieval" has anything to do with the issue at hand. The rest refer to the generative response itself, not the knowledge used. Evaluator / What it measures Document Retrieval: Whether the correct documents/chunks are retrieved and ranked appropriately (makes most sense here) Coherence: Whether the generated answer is logically consistent and understandable Groundedness: Whether claims in the answer are supported by retrieved evidence Relevance: Whether the generated response addresses the user's query Response Completeness: Whether the final response covers all required aspects of the question
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.