Automating GenAI Evaluation in CI Pipelines

AI-300 By islever Editorial Last updated

Configure evaluation and validation for generative AI applications and agents

Short answer

D Configure a GitHub Actions workflow that executes the runs to automate evaluations as part of CI.

Question

A team develops and manages a conversational assistant by using Microsoft Foundry.
The team requires generative AI to automatically evaluate every pull request of an agentic application and fail the build if safety thresholds are exceeded.
You need to automate evaluations as part of CI.
What should you configure?
  1. Blocklist applied to the model endpoint
  2. Content filter configured in warning mode
  3. Retrieval chunking strategy
  4. GitHub Actions workflow that executes the runs Correct Answer

Key Takeaways

The question tests the distinction between runtime safety filters and CI/CD gates; only GitHub Actions can enforce build failures based on evaluation thresholds.

This page explains how to implement automated generative AI evaluations in a CI pipeline using GitHub Actions workflows.

Common mistake: Choosing content filters (B) because they handle safety, but missing that they run at inference time rather than as a CI build gate.

Expert Analysis

The requirement is to automate evaluations as part of CI (Continuous Integration) for a pull request. In the Microsoft ecosystem, GitHub Actions is the standard CI/CD engine. To integrate evaluation into this flow, you must configure a GitHub Actions workflow that triggers on pull requests and executes the evaluation runs.

Options A and B describe runtime safety controls. While essential for protecting users in production, they do not provide a pre-deployment gate within the CI pipeline. Option C relates to data processing for RAG, which is unrelated to automated quality assurance or build gating. Only Option D provides the mechanism to fail the build based on evaluation results.

Official Reference

https://learn.microsoft.com/en-us/azure/ai-studio/how-to/evaluate-generative-ai-applications

This link opens the official vendor documentation in a new tab.

Exam Strategy

When a question specifies 'CI', 'pull request', or 'build failure', look for CI/CD integration tools like GitHub Actions. Do not select runtime security features like content filters or blocklists, as these operate during inference, not during the software development lifecycle.

Frequently Asked Questions

Can I use Azure ML pipelines instead of GitHub Actions?

Yes, but the question specifically asks about automating evaluations for pull requests, which is a CI task typically handled by GitHub Actions.

What is the difference between a blocklist and an evaluation?

A blocklist is a static filter for specific terms, while an evaluation uses models to assess safety and quality dynamically against defined thresholds.

More AI-300 FAQ →

Community Votes

D
100%

100% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 2 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 2

  1. RICHARDALEX007 👍 2 Answered D
    Automatically evaluating every pull request and failing the build when safety thresholds are exceeded is a CI task implemented as a GitHub Actions workflow that runs Foundry evaluations. Blocklist (A), content filter (B) and chunking (C) are runtime controls, not CI gates.
  2. exam_upskill 👍 1 Answered D
    D. GitHub Actions workflow that executes the runs

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Evaluating RAG Answer Quality with Measurable Techniques

S-grade

Azure ML Asset Registration for Reusability

S-grade

Assessing and Improving RAG Retrieval Quality Sequence

S-grade

Implementing RAG Flow with Microsoft Foundry and Azure OpenAI

S-grade

← Back to AI-300 Study Guide