How Should You Configure Foundry Multi-Agent Prompt Evaluations?
Short answer
Detect skipped evidence gathering with prompt adherence evals, validate handoff payloads with schema-match evals, and test in-scope responses plus out-of-scope refusals with responsibility-boundary evals.
Question
You have a Microsoft Foundry ticket-triage solution that uses connected agents. Each subagent prompt includes instructions for allowed tools and a JSON handoff.
You need to add automated prompt evaluations. The solution must identify changes that cause the subagents to do the following:
Skip mandated evidence gathering.
Return payloads that downstream agents cannot process.
Handle work outside their assigned responsibilities.
How should you configure the evaluation suite? To answer, select the appropriate options in the answer area.
NOTE: Each correct selection is worth one point.
Key Takeaways
You must match each regression type to the evaluator that measures it — tool/format drift is a prompt-adherence and schema-validity problem, while scope creep is a responsibility-boundary problem, not a formatting one.
In a Microsoft Foundry connected-agent ticket-triage solution, the evaluation suite must be mapped so each mandated behavior has a matching evaluator: prompt adherence catches subagents that skip required evidence gathering, schema-match validation catches JSON handoff payloads downstream agents cannot parse, and a responsibility-boundary evaluation catches subagents that drift outside their assigned scope. The correct configuration pairs each failure mode with the evaluator that measures exactly that dimension.
Common mistake: Learners apply a single generic quality evaluator (such as groundedness, fluency, or task completion) to all three failure modes, or they try to detect out-of-scope work with a schema/format check. Groundedness scores factual support, and schema checks only validate structure, so neither detects scope drift or skipped mandated steps.
Expert Analysis
Why the Answer Is Correct
Because each subagent prompt explicitly mandates allowed tools and a JSON handoff contract, the suite must test those three contracts separately. A prompt-adherence evaluation detects when a subagent answers without performing the evidence-gathering steps the prompt requires, which is a regression in instruction following rather than in answer quality. A schema-match evaluation validates each handoff payload against the expected JSON structure, so any change that produces output downstream connected agents cannot deserialize is flagged. An agent responsibility-boundary evaluation, which tests in-scope responses together with out-of-scope refusals, is the only evaluator that measures whether a subagent stays inside its assigned responsibilities, exactly as the first comment describes: "Agent responsibility boundaries => Test in-scope responses and out-of-scope refusals."Why Other Candidate Answers Are Wrong
A task-completion or conversation-quality evaluator (groundedness, fluency, coherence, relevance) scores whether the final answer sounds supported and readable, so a subagent can skip evidence gathering, emit a broken payload, or work out of scope and still score well. A tool-call-accuracy check alone only inspects the calls that were actually made and their parameters; it does not reliably catch a completely missing mandatory step or a non-processable handoff body, so it cannot cover all three requirements. A pure JSON/schema validation applied to responsibility scoping is category-mismatched — structure validation cannot express business boundaries, which is why scope drift needs in-scope/out-of-scope refusal testing instead.Community Comment Notes
Only one comment was posted, and it targets the scoping row directly: responsibility boundaries "concern what an agent is allowed to do, not how it formats its output," so the correct selection tests in-scope responses and out-of-scope refusals. That reasoning also implicitly rules out placing a format-oriented evaluator in the out-of-scope slot, which is the most common mis-selection for this hotspot. No comment addresses the evidence-gathering or JSON handoff rows, so those two mappings must be derived from the Foundry evaluator definitions rather than from community consensus.Official Reference
https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-evaluators/agent-evaluators
This link opens the official vendor documentation in a new tab.
Exam Strategy
Treat each hotspot row as a separate contract in the agent prompts — a behavioral instruction, a data-format instruction, and a scope instruction — and pick the evaluator that measures that contract type. Avoid any evaluator whose stated purpose is general answer quality; those never detect contract violations.
Frequently Asked Questions
Which evaluator catches a Foundry connected agent that skips mandated evidence gathering?
How do you detect when a subagent's JSON handoff payload cannot be processed downstream?
Can groundedness or task-completion evaluators detect out-of-scope agent work?
Discussion 1 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 1
-
95624c6f 👍 1Agent responsibility boundaries => Test in-scope responses and out-of-scope refusals Responsibility boundaries concern what an agent is allowed to do, not how it formats its output. Testing in-scope responses and out-of-scope refusals checks whether the agent accepts tasks within its responsibilities and refuses tasks outside them.
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.