Google Professional Data Engineer: The GCP Cert That Proves You Can Build Data Pipelines at Cloud Scale

Batch vs stream and the lambda/kappa architecture decision

The foundational design question in every data engineering role: when does the use case require stream processing, and when is batch sufficient? The PDE exam tests this judgment through scenario questions that encode the answer in requirements candidates must parse carefully.

• Batch processing: use when the use case tolerates latency measured in hours or days. Dataproc (managed Spark/Hadoop) is correct for legacy workloads already expressed as Spark jobs or when the team has deep Spark expertise. BigQuery scheduled queries and BigQuery Data Transfer Service handle periodic batch loads without custom pipeline code. Cloud Composer (managed Apache Airflow) orchestrates multi-step batch workflows across GCP services and external systems.
• Stream processing: use when the use case requires results in seconds to minutes. Pub/Sub ingests event streams from any source at any volume with push and pull subscriber models. Dataflow (Apache Beam) processes Pub/Sub streams with stateful windowing — tumbling windows for fixed time intervals, sliding windows for rolling aggregations, session windows for user activity tracking. Exactly-once semantics are a Dataflow differentiator: the Beam runner guarantees each message affects the pipeline state exactly once, even in the presence of failures and retries.
• Lambda architecture: two parallel paths — a batch layer for high-accuracy historical processing and a speed layer for low-latency approximations — merged at the serving layer. Operationally complex because you maintain two codebases producing the same output. Correct when you need both historical reprocessing and real-time output and you can tolerate the operational overhead.
• Kappa architecture: a single streaming pipeline that handles both real-time and historical processing. Simpler to operate than lambda because there is one codebase. Correct when Dataflow can replay historical events from a Pub/Sub Lite subscription or Cloud Storage archive at the same throughput as real-time events — and for most modern GCP-native data platforms, it can. The PDE exam strongly favors kappa recommendations in 2026 scenarios.
• Dataflow vs Dataproc selection: Dataflow is the PDE-preferred choice for new streaming pipelines because of auto-scaling, serverless operation, and Apache Beam portability. Dataproc is correct when migrating existing Hadoop/Spark workloads, when the team needs specific Spark libraries unavailable in Beam, or when the job runs on a fixed schedule against data already in Cloud Storage where Dataproc’s per-second cluster billing is cheaper than Dataflow’s throughput billing.

Designing for reliability and cost efficiency

• Fault tolerance patterns: dead-letter topics (DLT) in Pub/Sub route messages that cannot be processed after N delivery attempts to a separate topic for manual inspection or reprocessing. Checkpointing in Dataflow persists intermediate pipeline state so that a worker failure causes only recent work to be reprocessed, not the entire job from the start. BigQuery’s transactional writes (via Storage Write API) enable atomic multi-row commits for idempotent pipeline outputs.
• Autoscaling design: Dataflow’s Streaming Engine auto-scales workers based on backlog and processing rate. The PDE exam tests when autoscaling is insufficient: pipelines with state that cannot be rebalanced across workers (shuffle-heavy joins on unbounded data), or pipelines with downstream dependencies that cannot absorb sudden output bursts.
• Cost optimization at design time: BigQuery partitioned tables by ingestion time or a date column reduce bytes scanned per query. Clustered tables co-locate rows with the same cluster column values — correct when WHERE clauses frequently filter on the same high-cardinality column. Preemptible VMs on Dataproc clusters reduce cluster cost by 60–80% for fault-tolerant batch jobs. Pub/Sub Lite provides lower-cost messaging with zonal availability constraints — correct when messages are high-volume and loss of a zone is acceptable.

Pub/Sub subscriber patterns and pipeline integration

Pub/Sub is the entry point for event-driven data architectures on GCP. The PDE exam tests Pub/Sub configuration decisions that depend on downstream processing requirements.

• Push vs pull subscriptions: push delivers messages to an HTTPS endpoint — correct when the subscriber is a Cloud Run or App Engine service that scales to zero and should only wake on message arrival. Pull requires the subscriber to actively request messages — correct for Dataflow pipelines, which maintain persistent connections and process messages continuously. Never use push for a Dataflow pipeline: the push delivery model cannot match Dataflow’s per-worker throughput.
• Ordering and deduplication: Pub/Sub message ordering is disabled by default. Enable message ordering by setting an ordering key — all messages with the same key are delivered in publish order to a single subscriber. Deduplication relies on Dataflow’s exactly-once semantics via message ACK IDs; Pub/Sub itself does not deduplicate.
• Pub/Sub Lite vs standard Pub/Sub: standard Pub/Sub is globally replicated, zonal-failure-tolerant, and priced per-message-byte. Pub/Sub Lite is zonally replicated, lower-cost, and priced by provisioned throughput capacity. Choose Lite when throughput is predictable and high and you can tolerate zonal failure risk. Choose standard for mission-critical streams where zonal availability is a requirement.
• Dataflow + Pub/Sub pipeline pattern: Pub/Sub source → Dataflow PTransforms (parse, enrich, filter, aggregate) → BigQuery Storage Write API sink. The Storage Write API is the correct BigQuery sink for streaming Dataflow pipelines because it supports exactly-once writes and streams directly to BigQuery storage without the 1-minute buffer of the legacy streaming insert API.

Batch ingestion: Cloud Storage, Transfer Service, and Data Fusion

• Storage Transfer Service: managed data transfer from AWS S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources to Cloud Storage. Correct when migrating petabytes of data from another cloud provider without writing custom code. Supports scheduled transfers, bandwidth throttling, and object filtering by prefix, size, and last-modified date.
• BigQuery Data Transfer Service (DTS): managed connectors for SaaS data sources (Google Ads, YouTube, Salesforce, Oracle, Teradata) and cross-region BigQuery dataset copies. Correct when the data source is a supported SaaS application or when you need automated scheduled loads into BigQuery without pipeline code.
• Cloud Data Fusion: fully managed, code-free ETL built on Apache CDAP. Provides a visual pipeline builder with pre-built connectors to 150+ sources and sinks. Correct when business analysts or data engineers with limited coding background need to build and maintain pipelines, or when an organization is migrating SSIS or Informatica pipelines to GCP. Not correct when you need streaming processing, advanced transformations, or integration with Dataflow’s Apache Beam ecosystem.
• Dataproc for batch ETL: submit Spark or Hadoop jobs against data in Cloud Storage. Correct for organizations with large catalogs of existing Spark transformation code, complex multi-stage batch jobs, or workloads requiring specific Spark ecosystem libraries (Delta Lake, Apache Iceberg, Spark ML). Use ephemeral clusters that are created per-job and deleted after completion to minimize cost — the PDE exam consistently expects candidates to recommend ephemeral clusters over persistent clusters for batch workloads.

Storage service selection matrix

The most important domain-3 skill is distinguishing GCP’s storage services from each other. The exam routinely presents distractors that are wrong only because of a single requirement the question encodes.

• BigQuery: serverless, columnar, petabyte-scale analytics data warehouse. Correct when the use case is SQL analytics, reporting, or ML training on large datasets. Not correct for transactional writes with point lookups by primary key (BigQuery’s DML is batch-oriented; row-level reads require full table scans unless the table is clustered).
• Cloud Bigtable: managed, wide-column, HBase-compatible store. Correct for high-throughput, low-latency workloads with predictable access patterns: time-series data, IoT telemetry, financial tick data, user activity logs. Scales to petabytes with single-digit millisecond reads and writes. Not correct for SQL analytics, ad-hoc queries, or workloads with unpredictable access patterns — Bigtable has no query optimizer and performance is entirely dependent on row key design.
• Firestore: managed, document-oriented, serverless database. Correct for mobile/web application backends requiring real-time synchronization, strong consistency at document level, and automatic scaling from zero. Not correct for analytics workloads, complex joins, or data warehouse use cases.
• Cloud Spanner: globally distributed, strongly consistent relational database. Correct when the use case requires ACID transactions across multiple regions (global financial systems, global inventory management), SQL query support, and scale beyond what Cloud SQL can handle. The correct answer when the scenario specifies: “globally consistent,” “multi-region writes,” or “cannot tolerate eventual consistency.” Not correct for analytics workloads or when the use case fits within Cloud SQL’s scale limits.
• Cloud SQL: managed MySQL, PostgreSQL, or SQL Server. Correct for standard relational workloads, OLTP applications, and existing workloads that require a specific database engine. Not correct when global consistency across multiple regions is required (use Spanner) or when scale exceeds Cloud SQL’s limits (up to 128 vCPU and 864 GB RAM per instance).
• Cloud Storage: object storage, the data lake layer. Correct for staging data between pipeline stages, storing raw source files, archiving, and as the source/sink for Dataproc and Dataflow jobs. Storage class selection: Standard for frequently accessed data, Nearline for access once per month, Coldline for access once per quarter, Archive for retention with access once per year. The PDE exam tests Autoclass — automatic storage class management that moves objects to cheaper classes when access frequency drops, eliminating the need to manually set storage class policies.

BigQuery optimization and access control

• Partitioning strategy: partition BigQuery tables by ingestion time (_PARTITIONTIME), a DATE/TIMESTAMP column, or an INTEGER range. Queries with a WHERE clause on the partition column scan only the relevant partitions — reducing bytes billed directly. Partition pruning is the single highest-impact BigQuery cost optimization for time-series data. Require partition filter on tables with the require_partition_filter option to prevent full-table scans.
• Clustering: cluster a table by up to four columns. BigQuery co-locates rows with identical cluster column values in the same storage blocks. Queries that filter on cluster columns reduce bytes scanned beyond partition pruning. Clustering complements partitioning — partition first by time, then cluster by the most frequently filtered dimension (e.g., user_id, product_category, region). Unlike partitioning, clustering is not visible in the query plan until data is loaded — cluster cardinality matters: low-cardinality columns cluster effectively.
• Materialized views: BigQuery materialized views pre-compute and cache query results. The materialized view is automatically refreshed when the base table changes. Correct when the same aggregation query is executed frequently by many users — materialized views answer the query from cache rather than re-scanning the base table. The exam tests when materialized views are not appropriate: queries with user-specific filters that change the result per caller cannot be served from a shared materialized view.
• Authorized views and row-level security: authorized views expose a subset of a table’s columns to users who do not have access to the underlying table. Row-level access policies restrict which rows a user can see based on their identity or group membership. Correct when the question describes a shared analytics environment where different teams must see different slices of the same dataset without physical table duplication.
• BigQuery ML: train ML models in BigQuery using SQL syntax without exporting data to Vertex AI. Correct for logistic regression, linear regression, k-means clustering, ARIMA time-series forecasting, and importing TensorFlow models for batch scoring. Not correct for deep learning, complex custom model architectures, or online serving — those require Vertex AI.

Data governance: Data Catalog, Dataplex, and Looker

• Data Catalog: fully managed, scalable data discovery and metadata management service. Automatically catalogs BigQuery, Cloud Storage, Pub/Sub, and Dataproc metadata. Supports custom tags for business metadata (data owner, PII sensitivity classification, last verified date). Correct when the question asks for a solution to data discovery, lineage tracking, or business metadata management across a multi-project GCP environment.
• Dataplex: unified data management platform that provides data lakes, data quality rules, and automated governance across Cloud Storage and BigQuery assets. Dataplex data quality rules run as scheduled scans that validate column-level constraints (null checks, range checks, referential integrity) and report results to BigQuery. Correct when the question describes a data quality program that needs to scale across hundreds of datasets without per-dataset custom pipeline code.
• Looker and Looker Studio: Looker is the enterprise BI platform (semantic model-based, LookML language, governed metrics). Looker Studio (formerly Data Studio) is the self-service reporting tool for individual analysts. PDE candidates must distinguish them: Looker is correct when the question requires centralized metric governance, role-based data access, or embedded analytics in a third-party application. Looker Studio is correct for individual dashboards and quick visualizations without metric governance requirements.

Orchestration with Cloud Composer and monitoring

• Cloud Composer: managed Apache Airflow service on GCP. Correct when the use case requires complex multi-step workflow orchestration with conditional branching, retry logic, cross-service dependencies, and scheduling. PDE candidates must understand Directed Acyclic Graph (DAG) design: tasks as nodes, dependencies as edges, no cycles. Composer 2 uses GKE Autopilot for worker auto-scaling — workers scale to zero when no DAG runs are active, reducing idle cost compared to Composer 1’s fixed-size worker pools.
• Dataflow monitoring: Cloud Monitoring dashboards for Dataflow pipeline metrics: element count, system lag (how far behind real-time the pipeline is), worker CPU utilization, and job step throughput. System lag is the critical streaming health metric — growing system lag indicates the pipeline cannot process messages as fast as they arrive and needs more workers or a higher parallelism setting. Set up alerts on system lag exceeding a threshold before the Pub/Sub backlog exceeds subscriber quotas.
• BigQuery monitoring and quotas: BigQuery audit logs (data access logs and admin logs) are the correct answer when the question asks how to identify who queried what data and when — essential for compliance and for diagnosing unexpected cost spikes. BigQuery slot monitoring identifies projects consuming disproportionate reservation capacity. Set BigQuery reservation commitments for predictable workloads and use on-demand billing only for variable or exploratory queries.
• Cost controls: BigQuery dataset cost controls include per-user byte quotas (prevent runaway queries from a single analyst), project-level byte quotas, and BI Engine reservations for sub-second dashboard queries that eliminate per-query scan costs. Dataproc cost control: preemptible secondary workers for fault-tolerant batch jobs, ephemeral clusters terminated after job completion, and cluster scheduled deletion for jobs that run longer than expected.

Disaster recovery and multi-region design

• BigQuery multi-region datasets: create datasets in the US multi-region or EU multi-region location for geographic redundancy. Multi-region datasets replicate data across at least two regions within the multi-region. RPO is near-zero; RTO is minutes. The trade-off: cross-region query execution may route to the non-primary region, introducing latency variability. Use multi-region when data residency regulations permit and when disaster recovery SLAs require sub-hour RTO.
• Cloud Storage dual-region and multi-region: dual-region buckets replicate objects across two specific regions; multi-region buckets replicate across geographically separated regions within a continent. Turbo Replication (for dual-region) guarantees replication within 15 minutes — correct when the exam specifies an RPO of 15 minutes or less for Cloud Storage objects used as a data lake layer.
• Dataflow job recovery: Dataflow streaming jobs with snapshots enabled can be stopped and resumed from a snapshot, preserving in-flight state and the position of Pub/Sub subscription cursors. Correct when the question asks how to update a streaming pipeline’s code without data loss — take a snapshot, launch the updated job from the snapshot, then cancel the old job.