How Should You Route Failed Dataflow Messages for Alerting?

PDE By islever Editorial Last updated

Short answer

B Use an exception handling block in your Dataflow’s DoFn code to push the messages that failed to be transformed through a side output and to a new Pub/Sub topic. Use Cloud Monitoring to monitor the topic/num_unacked_messages_by_region metric on this new topic.

Question

Your car factory is pushing machine measurements as messages into a Pub/Sub topic in your Google Cloud project. A Dataflow streaming job, that you wrote with the Apache Beam SDK, reads these messages, sends acknowledgment to Pub/Sub, applies some custom business logic in a DoFn instance, and writes the result to BigQuery. You want to ensure that if your business logic fails on a message, the message will be sent to a Pub/Sub topic that you want to monitor for alerting purposes. What should you do?
  1. Enable retaining of acknowledged messages in your Pub/Sub pull subscription. Use Cloud Monitoring to monitor the subscription/num_retained_acked_messages metric on this subscription.
  2. Use an exception handling block in your Dataflow’s DoFn code to push the messages that failed to be transformed through a side output and to a new Pub/Sub topic. Use Cloud Monitoring to monitor the topic/num_unacked_messages_by_region metric on this new topic. Correct Answer
  3. Enable dead lettering in your Pub/Sub pull subscription, and specify a new Pub/Sub topic as the dead letter topic. Use Cloud Monitoring to monitor the subscription/dead_letter_message_count metric on your pull subscription.
  4. Create a snapshot of your Pub/Sub pull subscription. Use Cloud Monitoring to monitor the snapshot/num_messages metric on this snapshot.

Key Takeaways

This question tests your knowledge of Apache Beam error handling patterns, with the common trap being the assumption that managed Pub/Sub features like dead-letter topics integrate directly with Dataflow.

When processing Pub/Sub messages in Apache Beam pipelines, failed records must be routed using DoFn side outputs rather than relying on native Pub/Sub dead-letter topics. The community and official guidance consistently validate this pipeline-native approach for reliable monitoring and production alerting.

Common mistake: Option C is frequently chosen because candidates assume Pub/Sub dead-letter topics work seamlessly with Dataflow, but this configuration is officially unsupported and requires explicit pipeline implementation instead.

Expert Analysis

Why the Answer Is Correct

Using an exception handling block inside the DoFn allows you to catch transformation failures and route problematic messages via a side output to a dedicated Pub/Sub topic. This pipeline-native pattern aligns with Apache Beam best practices and enables precise Cloud Monitoring alerts on unacked message counts. By isolating errors at the processing stage, you maintain data integrity while ensuring real-time observability.

Why the Other Options Are Wrong

Option A monitors retained acknowledged messages, which does not isolate processing failures or trigger targeted failure alerts. Option C attempts to use Pub/Sub dead-letter topics, but Dataflow explicitly recommends against this due to architectural incompatibilities with its distributed execution model. Option D relies on snapshots, which are designed for state replay and debugging rather than real-time failure monitoring workflows.

Community Comment Notes

Community discussions heavily favor Option B, with users noting that explicit DoFn exception handling and side outputs effectively isolate failed messages for monitoring [1][4]. Several candidates questioned Option C until referencing official documentation that clarifies Pub/Sub dead-letter topics are unsupported with Dataflow pipelines [3]. The unanimous voting pattern confirms that examiners expect pipeline-level error management over external service configurations.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

Always differentiate between platform-managed features and SDK-level implementations when questions combine multiple Google Cloud services. For Dataflow scenarios, default to pipeline-native constructs like side outputs and explicit error handling unless the documentation explicitly confirms cross-service integration.

Community Votes

B
100%

100% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 8 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 8

  1. raaad 👍 14 Answered B
    - Exception Handling in DoFn: Implementing an exception handling block within DoFn in Dataflow to catch failures during processing is a direct way to manage errors. - Side Output to New Topic: Using a side output to redirect failed messages to a new Pub/Sub topic is an effective way to isolate and manage these messages. - Monitoring: Monitoring the num_unacked_messages_by_region on the new topic can alert you to the presence of failed messages.
  2. chrissamharris 👍 2
    Option C - dead letter topic is built in and requires no changes https://cloud.google.com/pubsub/docs/handling-failures Enable dead lettering in your Pub/Sub pull subscription, and specify a new Pub/Sub topic as the dead letter topic. Use Cloud Monitoring to monitor the subscription/dead_letter_message_count metric on your pull subscription.
  3. Jeyaraj 👍 1
    Option B. Here's why: Side Output for Failed Messages: Dataflow allows you to use side outputs to handle messages that fail processing. In your DoFn , you can catch exceptions and write the failed messages to a separate PCollection . This PCollection can then be written to a new Pub/Sub topic. New Pub/Sub Topic for Monitoring: Creating a dedicated Pub/Sub topic for failed messages allows you to monitor it specifically for alerting purposes. This provides a clear view of any issues with your business logic. topic/num_unacked_messages_by_region Metric: This Cloud Monitoring metric tracks the number of unacknowledged messages in a Pub/Sub topic. By monitoring this metric on your new topic, you can identify when messages are failing to be processed correctly.
  4. scaenruy 👍 1 Answered B
    B. Use an exception handling block in your Dataflow’s DoFn code to push the messages that failed to be transformed through a side output and to a new Pub/Sub topic. Use Cloud Monitoring to monitor the topic/num_unacked_messages_by_region metric on this new topic.
  5. 7787de3 👍 1 Answered B
    See here: https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#unsupported-features It's not recommended to use Pub/Sub dead-letter topics with Dataflow (...) Instead, implement the dead-letter pattern explicitly in the pipeline
Show 3 more comments
  1. joao_01 👍 1
    I would like to know why isn't anyone considering the option C.
  2. Matt_108 👍 1 Answered B
    Option B - Raaad explanation is complete
  3. hanoverquay 👍 1 Answered B
    option B

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to establish consistent BigQuery spend?

S-grade

Triggering Airflow DAGs Reactively in Private Cloud Composer

S-grade

How to resolve Cloud Composer 2 worker pod evictions and task failures?

S-grade

Which Google Cloud database for on-prem PostgreSQL with analytics?

S-grade

← Back to PDE Study Guide