Streaming Data Pipeline Design with Dataflow and Pub/Sub

The scenario requires a streaming data pipeline with low latency and high throughput, specifically involving filtering, enriching, and aggregating real-time player activity from Pub/Sub into BigQuery. Google Cloud's recommended service for serverless, scalable stream processing is Dataflow. Dataflow natively integrates with Pub/Sub via Apache Beam, allowing for efficient windowing, late-data handling, and direct streaming writes to BigQuery using the streaming API.

Option B is correct because Dataflow is designed for this exact pattern: it reads directly from Pub/Sub, performs complex transformations (filtering, enrichment, aggregation) in-memory with high efficiency, and writes to BigQuery. It autoscales automatically to handle throughput spikes common in gaming scenarios.

Option A (Cloud Composer) is incorrect because it is an orchestration tool, not a stream processor. While it can trigger workflows, it does not provide the low-latency, continuous processing required for real-time streaming data; it is better suited for batch or scheduled tasks.

Option C (Dataproc) involves managing Spark clusters. While Spark Structured Streaming can process data from Pub/Sub, it requires significant cluster management overhead and is generally less cost-effective and more complex to operate than the fully managed Dataflow for this specific use case. Google recommends Dataflow over Dataproc for pure streaming analytics unless specific custom Spark libraries are needed.

Option D (Cloud Run functions) is unsuitable for high-throughput streaming due to cold starts, concurrency limits, and the lack of native stateful aggregation capabilities found in stream processing engines like Beam/Dataflow. It is better for event-driven, lightweight tasks rather than continuous high-volume data pipelines.