Data Engineering Interview Questions (2026)

A data pipeline is a set of steps that moves data from source systems to a usable target such as a warehouse table, dashboard, feature table, or operational tool.

In an interview, define the source, transformation logic, schedule, storage layer, tests, monitoring, and owner. A pipeline is not only code. It also includes contracts, rerun behavior, alerting, and user trust.

Data grain means what one row represents: one order, one order item, one customer per day, one event, or one account snapshot.

Grain is the base of data modeling. If you miss it, joins duplicate rows, metrics drift, and tests look correct while the output is wrong. The grain comes before primary keys, dimensions, or aggregations.

Choose batch when the business can wait for scheduled data and the workload benefits from simpler orchestration, lower cost, easier reconciliation, and easier backfills.

Choose streaming when the decision loses value if data arrives late: fraud checks, alerts, real-time personalization, operational monitoring, or live inventory. Do not pick streaming just because it sounds advanced.

ETL transforms data before loading it into the target system. ELT loads raw data first, then transforms it inside the warehouse or lakehouse.

A practical answer ties the choice to control and compute. ETL is useful when data must be cleaned before storage or the target is limited. ELT works well when cloud storage and warehouse compute can hold raw history and run repeatable transforms.

A warehouse stores structured, modeled data for analytics. A data lake stores raw or semi-structured files cheaply. A lakehouse adds warehouse-style tables, transactions, and governance on top of lake storage.

The interview answer should mention workload fit. Warehouses are strong for BI and SQL analytics. Lakes are useful for raw history, logs, and flexible processing. Lakehouses try to give teams both low-cost storage and managed table behavior.

Partitioning splits data by values such as date, region, or event type so queries and jobs can read only the relevant slices.

Good partitioning lowers scan cost and job time. Bad partitioning creates too many small files, hot partitions, or full scans. In interviews, the partition column connects to the common filter and the arrival pattern.

An idempotent pipeline can run again for the same input without creating duplicates, changing totals incorrectly, or corrupting the target.

Use stable keys, deterministic transforms, overwrite-by-partition, merge statements, or checkpoints. In interviews, say how a retry behaves after a task fails halfway through a load.

A full refresh rebuilds all target data. An incremental load processes only new or changed records since a known checkpoint, watermark, timestamp, or version.

Incremental loads cut cost and time, but they need a clean change signal and a backfill plan. Interviewers often ask about late-arriving records because timestamp-only filters miss updates that arrive out of order.

Change data capture, or CDC, captures inserts, updates, and deletes from a source system so downstream systems can stay in sync without scanning the whole source every time.

CDC is useful for large operational databases, near-real-time replicas, and audit trails. A strong answer mentions ordering, schema changes, delete handling, replay, and what happens when the connector falls behind.

Production pipelines need checks for freshness, schema, nulls, uniqueness, accepted values, row counts, referential integrity, ranges, and reconciliation with source totals.

The checks should match business risk. A customer table may need uniqueness and valid email checks. A payments table needs reconciliation and duplicate controls. A dashboard mart needs freshness and metric drift checks.

Data lineage shows where a dataset came from, what transformed it, and which downstream tables, dashboards, models, or syncs depend on it.

Lineage helps with impact analysis, audits, incident response, and trust. If a source column changes, lineage tells you what breaks. If a metric is wrong, lineage helps trace the fault back to the right stage.

An orchestrator defines task dependencies, schedules jobs, retries failures, tracks state, manages backfills, exposes logs, and shows pipeline status in one place.

Cron can run a script. It does not understand that one task must wait for another table, that a failed partition needs a retry, or that a backfill should run for a date range. Airflow models that workflow as a DAG.

Parquet is a columnar file format. It stores data by column, supports compression, and lets engines read only the columns needed for a query.

It is a strong fit for analytics because queries often scan a few columns across many rows. CSV is easier to inspect, but it lacks schema, type safety, compression, and column pruning.

A slowly changing dimension stores how descriptive attributes change over time, such as a customer's plan, region, or account manager.

Type 1 overwrites the old value. Type 2 keeps history by adding a new row with effective dates or current flags. Interviewers ask this because business reports often need to know what was true at the time of an event.

A star schema organizes data into fact tables for measurable events and dimension tables for descriptive context.

It makes BI queries easier, reduces metric confusion, and gives teams a stable model for dashboards. In interviews, the business process, define the fact grain, then attach dimensions such as customer, product, time, and region comes first.

A Kafka topic is a named stream of records. Partitions split that stream so data can be stored, read, and processed in parallel while preserving order inside each partition.

Ordering is not global across the topic. It is only guaranteed within a partition. A good answer explains the partition key, consumer group behavior, offsets, replay, retention, and what happens when a consumer lags.

A Spark shuffle moves data across executors so records with the same key can be grouped, joined, sorted, or aggregated together.

Shuffles are expensive because they move data over the network and write intermediate files. Reduce them by filtering early, selecting needed columns, using broadcast joins for small tables, and choosing partition keys carefully.

A data contract defines what a producer promises to send: schema, types, allowed values, freshness, meaning, ownership, and change process.

Contracts reduce silent breakage. If product removes a field, changes an event name, or changes timestamp meaning, downstream tables and dashboards can fail. A contract makes those changes visible before users see wrong data.

Personally identifiable information should be classified, minimized, masked or tokenized where possible, encrypted in storage and transit, and restricted by role-based access.

Also mention retention, audit logs, deletion requests, and separate raw zones from curated zones. A data engineer is often responsible for making the safe path easy, not only writing transformation code.

Data observability is the ability to detect, diagnose, and explain data problems through signals such as freshness, volume, schema, distribution, lineage, and quality tests.

It turns pipeline failures and silent data drift into visible incidents. A strong interview coverage names alert thresholds, owners, severity, runbooks, and user communication when a trusted table is stale or wrong.

the dashboard metric: revenue by day, product, region, and channel comes first. Then define the order source, table grain, late-arriving order rule, quality checks, and refresh deadline.

A simple design is a daily batch pipeline: ingest orders, dedupe by order_id, join stable dimensions, write a partitioned fact table, run reconciliation against the source, then publish only if checks pass.

Inspect ownership, schema, primary keys, update and delete behavior, timestamp meaning, volume, rate limits, nulls, duplicates, and past incidents.

Do not start by writing the DAG. Start by proving whether the source can support the target table. Ask who owns the source, whether backfills are allowed, and how changes are announced.

It needs a source filter, checkpoint or watermark, staging table, merge into the target, data quality checks, and a way to rerun a time range safely.

Use a small overlap window if records can arrive late. Store load metadata such as run_id, loaded_at, source_count, target_count, and checkpoint value so failures can be audited.

A good DAG has clear task boundaries, explicit dependencies, retries, sensible timeouts, alerts, parameterized dates, and idempotent task logic.

Do not put all logic inside one PythonOperator. Keep extract, transform, test, and publish steps visible. Use Airflow for orchestration and observability, not as a place to hide every line of business logic.

Handle schema change through contracts, validation, versioning, compatibility rules, and alerting. Do not let unexpected changes silently flow into trusted tables.

For additive fields, staging can usually accept the new column while curated models decide whether to expose it. For removed or renamed fields, fail fast or route records to quarantine until the owner confirms the change.

Run the backfill in controlled partitions or a separate target, validate counts and metrics, then swap or merge after checks pass.

Backfills should use the same transformation code as normal runs. If the logic changed, tag the version and document the metric impact. Protect current dashboards from half-filled tables.

Find the business key or event id, choose the winning record rule, and make the dedupe step deterministic.

For app events, dedupe by event_id when available. If not, use a composite key such as user_id, event_name, event_time, and source request id. Choose latest ingestion time only when it matches the business rule.

Set a clear lateness policy: overlap recent partitions, mark data freshness, and decide whether late events update closed periods or go into an adjustment table.

The answer depends on the business. Finance may need locked periods with adjustments. Product analytics may accept recomputing the last few days. The important part is making the rule visible and repeatable.

Choose a column that matches common filters, arrival pattern, retention rules, and backfill needs. Date partitions are common because most analytics queries filter by time.

Avoid high-cardinality partitions such as user_id when they create too many small pieces. Use clustering or sorting for common secondary filters where the platform supports it.

Check scan volume, partition pruning, join explosion, repeated subqueries, changed filters, statistics, and whether an upstream table got much larger.

Use the query plan or job history. The fix might be partition filters, pre-aggregation, removing unnecessary columns, materializing a repeated step, or correcting a bad join key.

Check the Spark UI for stage time, shuffle read and write, skewed tasks, failed tasks, spill, input size, partition count, and join strategy.

Then reduce data early, select fewer columns, fix skew, broadcast small dimensions, tune partitions, and write fewer larger files. Avoid guessing from code alone.

Check whether producers sped up, consumers slowed down, partitions are uneven, processing time increased, broker health changed, or downstream writes are throttled.

Short-term fixes may include scaling consumers up to the partition count, reducing per-message work, batching writes, or pausing non-critical consumers. Long-term fixes need metrics on throughput and processing time.

Ingest the ordered change stream, stage raw events with operation type and log position, then apply inserts, updates, and deletes into the target with deterministic ordering.

Keep the raw CDC history for replay. Handle schema versions, tombstones, out-of-order events, and connector restarts. The warehouse table should be the current state or a history table, depending on the use case.

Use dbt tests after transformations to prove assumptions such as uniqueness, not-null fields, accepted values, relationships, and business rules.

Tests should protect trusted models, not only raw staging. The direct answer is what happens when a test fails: stop publish, alert the owner, and keep stale data marked rather than silently replacing it.

Compare the current dimension row with the incoming source row. If tracked attributes changed, close the old row and insert a new row with a new valid_from date.

Use a stable customer_id, effective dates, current flag, and a hash of tracked attributes. Make the process idempotent so reruns do not create extra history rows.

Store expected update times and compare them with the latest successful load time, latest source event time, and downstream publish time.

Freshness alerts includes severity and owner. A sales dashboard missing by ten minutes may be low severity. A fraud feature table stale by ten minutes may be urgent.

Control scan volume, compute size, scheduling, storage lifecycle, repeated transformations, small files, retention, and user query patterns.

Cost answers should be specific. Partition large tables, materialize common marts, avoid repeated full refreshes, right-size warehouses or clusters, compact files, and set budgets or alerts.

Document the metric definition, source tables, transformation logic, grain, filters, owner, freshness target, and known caveats.

Then link the dashboard to the curated model, not to a hidden SQL copy. Use lineage tooling where available, but The technical detail also include plain-language ownership and definitions.

Classify the fields, restrict raw access, mask or hash where possible, encrypt data, log access, set retention, and avoid copying PII into wide analytics tables.

Build separate raw and curated zones. The curated table should expose only the fields needed by users. If downstream tools need contact attributes, use approved syncs with ownership and deletion support.

Respect rate limits, paginate safely, checkpoint pages or cursors, retry transient errors with backoff, and store raw responses before transformation.

Also handle schema drift, authentication expiry, duplicate pages, and vendor downtime. The technical detail is audit logs for request count, response count, and last successful cursor.

Stop the trusted publish path, alert the owner, confirm whether the meaning changed, and patch staging only after the contract is updated.

A fast alias may restore the pipeline, but it can hide a semantic change. Check row counts, joins, duplicates, and downstream metrics before reopening the table.

Work backward from the metric to the mart, transform, staging, and source totals. Compare row counts, changed records, filters, joins, and recent deploys.

Do not a code rewrite comes first. First establish whether the business changed, the source changed, or the pipeline changed. Then communicate whether the dashboard is trustworthy while the investigation runs.

Freeze publish for affected partitions, identify the duplicate key, compare with the gateway source, fix the dedupe logic, and republish after reconciliation passes.

Payments need stronger controls than many event streams. Use gateway transaction id where possible, keep an audit table of rejected duplicates, and communicate affected periods to finance.

Find the slow or unstable stage, set stage-level alerts, add capacity or scheduling changes, and create a runbook for the owner.

Also review whether the SLA is realistic. If the source lands at 6:55 AM and the transform takes 20 minutes, the 7 AM promise is impossible without changing the source, scope, or compute plan.

Inspect state size, window length, cardinality, late event policy, checkpoint growth, batch size, and sink backpressure.

Large state usually comes from long windows, high-cardinality keys, or late data kept too long. Reduce state, set watermarks, tune partitions, and confirm the sink can keep up.

Store raw payloads, extract stable fields into typed staging columns, version parsing logic, and alert when required fields disappear or change type.

Do not flatten every field into a trusted table immediately. Keep raw history for replay, then expose curated fields only when the contract and business meaning are stable.

Run the corrected logic in a separate target by partition, compare totals with the old output and finance source, then publish after approval.

Backfills change history. Keep old and new versions long enough to audit the difference, document the metric impact, and avoid rebuilding all partitions in one unmonitored job.

Use tenant-aware access controls, row filters, separate encryption or storage boundaries where needed, tests for tenant isolation, and strict review on shared models.

In interviews, explain both design and verification. A query filter alone is not enough if analysts can access raw shared tables. Add policy checks and automated tests using known tenant ids.

The pipeline must know where user data exists, delete or anonymize it according to policy, update derived tables, and keep audit evidence without retaining restricted values.

Lineage is critical here. If user_id appears in raw events, curated tables, feature stores, exports, and backups, the deletion workflow must cover each copy or define a compliant retention exception.

Expect questions about where exactly once applies: source reads, state updates, sink writes, retries, dedupe keys, and transactional guarantees.

A careful answer avoids overclaiming. Many systems provide exactly-once processing within part of the pipeline, but the sink or external side effect may still need idempotent writes.

Reconcile counts, sums, date ranges, primary keys, and status distributions between source and target at the same grain and cutoff time.

Use source snapshots or extraction audit logs so both sides compare the same time window. For money, reconcile amounts by currency, gateway, and date, not only a total row count.

Clarify users, grain, source priority, identity resolution, freshness, PII rules, and whether the output is for analytics, sales sync, ML, or product use.

A customer 360 table often joins CRM, billing, product events, support, and marketing data. The hard part is not the join. It is defining the customer entity and conflict rules across systems.

Inventory each script, input, output, schedule, dependency, owner, and failure behavior. Then migrate in small DAGs with the same outputs and better observability.

Do not rewrite all logic first. Wrap stable scripts, expose dependencies, add retries and alerts, then refactor business logic after the new orchestration path is proven.

Break cost down by job, user, warehouse, table, and schedule. Identify new workloads, query plan changes, larger inputs, and repeated full refreshes.

Then fix the cause: partition filters, incremental models, warehouse sizing, materialized marts, query review, budgets, or retention changes. Cost control should not silently remove data quality checks.

The consumer should validate messages, send bad records to a dead-letter path with error context, alert the owner, and continue processing valid records when safe.

Do not drop bad messages silently. Keep enough payload and metadata to debug, but redact sensitive fields. If the malformed rate crosses a threshold, fail the pipeline or page the producer owner.

Stop downstream publish, identify the bad run, restore the last good version or partition, fix the transform, rerun affected data, and document user impact.

Recovery depends on having versioned tables, partitioned writes, raw replay data, and audit logs. If those do not exist, say what you would add after the incident.

Check feature freshness, training-serving skew, source delays, pipeline failures, null spikes, and whether online and offline feature logic still match.

Data engineering for ML needs stronger freshness and consistency checks than many dashboards. Add feature-level freshness alerts and compare online serving values against offline backfills.

Provide metric definition, source systems, transformation code version, run logs, data quality checks, lineage, access records, and reconciliation outputs.

The technical detail sound boring and traceable. Audits punish mystery. A strong platform can explain what data existed, what code ran, when it ran, who had access, and what checks passed.

Add staging outputs, backward-compatible schema changes, contract tests, canary runs, downstream impact checks, and a rollback path.

Data releases need deployment discipline. Rename columns, grain changes, and metric changes should be versioned or announced before the old output disappears.

Frame it as an ownership trade-off. Central teams improve consistency and shared standards. Domain teams understand source meaning better. Most companies need both with clear contracts.

Avoid slogans. Say which team owns source events, which team owns shared marts, who approves metric definitions, and how quality rules are enforced across domains.