Troubleshooting Duplicate Ingestion in BigQuery Pipelines
Short answer
C Investigate duplicate rows, correlate BigQuery Audit logs and Cloud Monitoring data to identify concurrent pipeline versions, then stop older versions to fix the data increase.
Question
Key Takeaways
When data volume doubles post-deployment despite unchanged input, suspect concurrent old/new pipeline instances; use audit logs to verify and stop duplicates.
Learn how to diagnose and fix data duplication in BigQuery caused by ghost pipelines during deployment updates using Cloud Monitoring and Audit Logs.
Common mistake: Choosing rollback (D) or immediate deduplication (A) ignores the need to identify the root cause via logs, leading to recurring issues or wasted resources.
Expert Analysis
The scenario describes a classic "ghost pipeline" or duplicate ingestion issue following a deployment. The key constraints are that Pub/Sub volume remained constant, yet BigQuery partition sizes doubled for specific tables immediately after a new pipeline version was deployed. This strongly implies that the old pipeline instance was not successfully terminated or drained before the new one started, resulting in both versions writing to the same destination simultaneously.
Option C is the correct approach because it prioritizes investigation and root cause identification before remediation. Step 1 checks for the symptom (duplicate rows). Step 2 uses BigQuery Audit logs to identify the specific job IDs responsible for the writes. Step 3 correlates these jobs with Cloud Monitoring data to determine their start times and code versions, confirming if multiple versions ran concurrently. Step 4 provides the fix: stopping all but the latest valid version to prevent further duplication. This aligns with best practices for operationalizing pipelines, where understanding the lifecycle of jobs is critical.
Option A suggests deduplication as a primary fix, which treats the symptom rather than the cause and adds unnecessary compute cost. Option B skips the investigative correlation steps and jumps to restoration, which doesn't address the active bug causing double writes. Option D rolls back without investigating, potentially masking the deployment flaw, and replaying messages can lead to further complexity if the underlying logic error persists.
This question tests your ability to troubleshoot pipeline deployments using GCP observability tools (Audit Logs, Cloud Monitoring) and understand the operational consequences of improper pipeline version management.
Exam Strategy
Always correlate resource usage spikes with deployment timelines. When facing data anomalies post-deployment, check for concurrent job executions using BigQuery Audit Logs and Cloud Monitoring before applying fixes like deduplication or rollback.
Frequently Asked Questions
Why is checking BigQuery Audit logs important in this scenario?
What is a 'ghost pipeline' in GCP data engineering?
Why shouldn't I just deduplicate the data immediately?
Community Votes
60% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.
Discussion 9 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 9
-
raaad 👍 12 Answered C- Detailed Investigation of Logs and Jobs Checking for duplicate rows targets the potential immediate cause of the issue. - Checking the BigQuery Audit logs helps identify which jobs might be contributing to the increased data volume. - Using Cloud Monitoring to correlate job starts with pipeline versions helps identify if a specific version of the pipeline is responsible. - Managing multiple versions of pipelines ensures that only the intended version is active, addressing any versioning errors that might have occurred during deployment. ======= Why not B While it addresses the symptom (excess data), it doesn't necessarily stop the problem from recurring. (The questions asked to investigate and fix)
-
RenePetersen 👍 3This does not fix the error, it basically assumes that the error is not really there.
-
task_7 👍 2 Answered BB. Check for code errors in the deployed pipelines, multiple writing to pipeline BigQuery sink, errors in Cloud Logging, and if necessary, restore tables using time travel. Check for code errors Check for multiple writes Check Cloud Logging Restore tables if necessary:
-
SamuelTsch 👍 2 Answered BNo idea which one to choose. Option C miss a step - to restore the tables.
-
carbonscave 👍 1 Answered CWhy C is the right moveThe "Ghost Pipeline" Anti-Pattern: The symptoms in the prompt are classic indicators of a botched deployment. If the incoming Pub/Sub volume hasn't changed, but the BigQuery partition size exactly doubled on the day of a release, it almost guarantees that the deployment failed to drain or cancel the old pipeline. Both the old version and the new version of the pipeline are currently running concurrently, reading the same messages, and writing them twice. The Investigation Playbook: Option C follows the exact Google Cloud incident response path for rogue writers. You confirm the duplicates exist $\rightarrow$ you check BigQuery Audit Logs to capture the specific principalEmail or jobId writing the data $\rightarrow$ you trace that identity back to the active Dataflow jobs. The Root Cause Fix: Once you visually confirm that multiple versions of the pipeline are running simultaneously and targeting the same tables, stopping the older, orphaned versions immediately stops the duplication at the source.
Show 4 more comments
-
Ryannn23 👍 1" You need to investigate and fix the cause of the data increase. " - fixing the target tables was not required.
-
judy_data 👍 1Because option D doesn't investigate the root cause of the issue and doesn't even check the code.
-
Matt_108 👍 1 Answered COption C - agree with Raaad on the reasons
-
mi_yulai 👍 1Why not D?
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.