Mitigating Downstream Service Failures with Circuit Breakers

PCDOE By islever Editorial Last updated

Balancing change, velocity, and reliability of the service.

Short answer

C Implement a circuit breaker to store the request data in a database.

Question

You manage a microservice that provides a public-facing API (Service A). Service A is time-critical and has a response SLO of 500 ms. Service A makes synchronous calls to internal API (Service B) that is known to become unreliable under heavy load, resulting in connection timeout errors or 500 errors. Service B is used to collect request information for the transactions processed by Service A. You need to mitigate the impact of Service B issues on your Service A users. What should you do?
  1. Increase the size of the queue in front of the thread pool used by Service A instances.
  2. Implement retry logic with exponential back-offs when calling Service B.
  3. Implement a circuit breaker to store the request data in a database. Correct Answer
  4. Implement rate limiting in Service A to limit the number of requests to Service B.

Key Takeaways

The question tests the ability to isolate failures using resilience patterns like circuit breakers to maintain SLOs when downstream services degrade.

Learn how to protect a high-latency-sensitive API from unreliable dependencies using the circuit breaker pattern in Google Cloud architecture.

Common mistake: Choosing retry logic (B) is a common trap because retries seem helpful, but they often amplify load on a failing backend, worsening the outage and violating SLOs.

Expert Analysis

The scenario describes a classic dependency failure pattern where a synchronous call to an internal service (Service B) is causing latency and errors for the public-facing service (Service A). The primary goal is to mitigate the impact on Service A users, specifically preserving its 500 ms SLO. When Service B becomes unreliable, continuing to attempt synchronous calls consumes threads and time, leading to cascading failures or timeouts in Service A.

The correct approach is to implement a circuit breaker pattern. This design pillar of resilience engineering monitors the health of the downstream call. If Service B fails beyond a certain threshold, the circuit 'opens,' immediately failing fast without attempting further calls to Service B. This prevents Service A from wasting resources on doomed requests. To ensure data is not lost during this outage, the circuit breaker should allow Service A to store the request data asynchronously in a durable store (like a database or message queue) rather than blocking on the synchronous call. This isolates the failure and protects the user experience.

Option A (increasing queue size) does not address the root cause; it only delays the inevitable timeout or resource exhaustion. Option B (retry logic with exponential back-off) can exacerbate the problem by adding load to an already struggling Service B, potentially causing a denial-of-service effect. Option D (rate limiting) restricts traffic but does not handle the failure mode of Service B itself; it is a protective measure but not a mitigation for existing unreliability in the same way a circuit breaker provides isolation and graceful degradation.

Exam Strategy

When faced with questions about maintaining SLOs under degraded conditions, look for patterns that provide isolation and fail-fast behavior. Avoid solutions that increase load (like retries) on the failing component. Instead, prioritize mechanisms that prevent resource exhaustion, such as circuit breakers or bulkheads, which allow the system to degrade gracefully while preserving core functionality.

Frequently Asked Questions

Why not just retry Service B when it times out?

Retries can create a thundering herd effect, overwhelming an already unstable Service B. This increases latency for all users and can lead to complete service unavailability, violating the 500 ms SLO.

What is the role of the database in the circuit breaker pattern?

The database acts as a durable storage for requests that cannot be processed immediately due to the downstream failure. This ensures no data loss while the circuit is open, allowing for later processing once Service B recovers.

More PCDOE FAQ →

Discussion 4 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 4

  1. Tartaruga1234 👍 4 Answered C
    C is correct
  2. gaurav517 👍 2 Answered C
    C. Implement a circuit breaker to store the request data in a database. Isolation (Circuit Breaker): The circuit breaker pattern monitors calls to Service B. If the error rate (timeouts or 500 exceeds a threshold, the circuit opens and immediately prevents Service A from calling Service B. This prevents Service B's unreliability from consuming resources (threads) and latency budget in Service A. Graceful Degradation (Fallback): Instead of immediately returning a 500 error to the client (which would breach the SLO), Service A executes a fallback action. The most appropriate fallback is to store the request data in a reliable location (a database or message queue).
  3. BlackCoffee 👍 1 Answered C
    Implementing a circuit breaker isolates Service A from Service B's failures and store the data in a database.
  4. katayoonnt 👍 1 Answered B
    https://cloud.google.com/storage/docs/retry-strategy

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Submitted for moderation before publishing. Keep it helpful and respectful.

Error Budget Exhausted: Should You Deploy an Urgent Customer Feature?

S-grade

Choosing a Latency SLO from Measured p90/p95 Performance

S-grade

SRE Postmortem Triggers: External Stakeholders and Data Loss

S-grade

Centralizing Production Folder Logs with an Aggregated Log Sink

S-grade

Practice All PCDOE Questions

Access 75 questions with complete answers and detailed explanations.

View Full PCDOE Practice Test →

← Back to PCDOE Study Guide