Why Did BigQuery Daily Storage Increase 50% After Pipeline Deployment?
You are monitoring your organization’s data lake hosted on BigQuery. The ingestion pipelines read data from Pub/Sub and write the data into tables on BigQuery. After a new version of the ingestion pipelines is deployed, the daily stored data increased by 50%. The volumes of data in Pub/Sub remained the same and only some tables had their daily partition data size doubled. You need to investigate and fix the cause of the data increase. What should you do?
Community Votes
76% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
This question tests symptom-driven troubleshooting: unchanged source data plus doubled partition size strongly indicates duplicate rows, so the first action should be checking for duplicates in the affected BigQuery tables, not blindly rolling back.
When a new BigQuery ingestion pipeline causes daily stored data to jump while Pub/Sub volumes stay the same, duplicate writes are the prime suspect. Community consensus favors checking for duplicate rows in the affected partitions before considering rollback or broad code review.
A common wrong answer is B, checking for code errors, because the data increase followed a deployment. However, without first inspecting the affected tables, this is too broad and may waste time; code review or rollback should come after confirming duplicate rows.
Community Discussion (5 comments)
- Detailed Investigation of Logs and Jobs Checking for duplicate rows targets the potential immediate cause of the issue. - Checking the BigQuery Audit logs helps identify which jobs might be contributing to the increased data volume. - Using Cloud Monitoring to correlate job starts with pipeline versions helps identify if a specific version of the pipeline is responsible. - Managing multiple versions of pipelines ensures that only the intended version is active, addressing any versioning errors that might have occurred during deployment. ======= Why not B While it addresses the symptom (excess data), it doesn't necessarily stop the problem from recurring. (The questions asked to investigate and fix)
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Option C — checking for duplicate rows in the BigQuery tables with doubled partition data — directly addresses the symptom. A 50% storage increase with unchanged Pub/Sub volume means the source data size did not change, so the extra bytes likely come from the pipeline writing the same events more than once. The fact that only some tables doubled further supports a pipeline logic error that duplicates writes to certain partitions, not a global increase.Why the Other Options Are Wrong
Option B suggests checking for code errors, but that is a vague and broad action; you would not know what to look for without first confirming the data is duplicated. Option D, rolling back the deployment, is a drastic step and may not be necessary if the issue is limited to certain tables. Options A and C are textually identical in the prompt, but the intended answer is the duplicate-row check, which should then be followed by investigating logs/jobs and restoring tables if needed.Community Comment Notes
One commenter supporting C noted that checking BigQuery Audit logs and Cloud Monitoring can help identify which jobs are responsible, reinforcing that duplicate-row discovery is the right starting point. Another commenter pointed out that option C misses a step — restoring the tables after fixing the issue — but this does not make the initial check incorrect. A few users chose B, but as one commenter explained, a more complete workflow includes checking for multiple writes in the pipeline and using time travel to restore, which begins with detecting duplicates.Official Reference
Exam Strategy
When investigating storage increases in BigQuery, always compare source volumes with destination volumes first. Use a COUNT(*) GROUP BY on the suspected primary key or deduplication key in the affected partition to quickly confirm duplicates, then check audit logs or Cloud Monitoring to find the responsible pipeline job before deciding on rollback or restoration.