ADP — frequently asked questions

SQL in BigQuery is optimized for large-scale, distributed processing. It allows you to clean and analyze data in the same place, reducing data movement and latency compared to external Python scripts.

Yes, but it is typically reserved for complex, multi-step workflows requiring orchestration. For straightforward cleaning tasks, native BigQuery SQL is simpler and more cost-effective.

Yes. You can use the BashOperator in Apache Airflow to execute existing shell scripts directly within a DAG without needing to rewrite the logic.

Cloud Scheduler is good for simple time-based triggers but lacks the advanced orchestration features, dependency management, and detailed observability provided by Airflow in Cloud Composer.

A bucket retention policy applies to all objects in the bucket and enforces a mandatory minimum retention period globally. An object hold is applied to individual objects and requires manual intervention to remove.

No. Once a retention policy is set, no user (including project owners) can delete or overwrite objects in the bucket until the retention period expires.

While possible, it is not recommended. Cloud Monitoring is designed for metric-based threshold alerting, which is more reliable and efficient than parsing logs for keywords.

Built-in metrics include worker CPU utilization, job lag, throughput, and error counts, which can be directly used to create alerting policies.

Cloud Composer is an orchestrator, meaning it schedules and monitors jobs. It does not have built-in capabilities to continuously ingest and process streaming data in real-time. It is best used for batch workflows.

Dataproc might be chosen if you need to use specific open-source libraries not supported by Dataflow/Beam or if you require tight control over the Spark cluster configuration. However, for most standard streaming ETL tasks, Dataflow is more scalable and easier to manage.

The default model type for time series forecasting in BigQuery ML is ARIMA_PLUS, which automatically handles missing values, outliers, holidays, and multiple seasonality.

Linear regression can be used for simple forecasting if there are no significant seasonal or temporal trends. However, for most ecommerce demand scenarios with seasonality, ARIMA_PLUS via BigQueryML is more appropriate and accurate.