Spark Interview Questions (2026)
Practice 45 Spark interview questions on RDDs, DataFrames, Spark SQL, Catalyst, shuffles, joins, caching, partitioning, Structured Streaming, and tuning.
45 questions with answersWhat Is Apache Spark?
Key Takeaways
- ✓Apache Spark is an open-source engine for distributed data processing across batch, SQL, streaming, machine learning, and graph workloads.
- ✓Spark interviews test execution flow: driver, executors, jobs, stages, tasks, partitions, transformations, actions, and shuffles.
- ✓Strong answers explain why a Spark job is slow, not only which API to call.
- ✓DataFrame and Spark SQL answers should mention Catalyst, physical plans, partitioning, joins, caching, and file layout.
Apache Spark is a distributed processing engine used for large-scale batch jobs, SQL analytics, streaming pipelines, and machine learning workloads. In interviews, Spark questions test whether you understand how driver code becomes jobs, stages, and tasks, why shuffles are expensive, how joins are planned, and how to debug slow or failing applications.
Watch: Using Apache Spark to analyze open data
Video: Using Apache Spark to analyze open data (Databricks, YouTube)
Test yourself and earn a certificate
6 quick questions. Score 70%+ to download your Spark certificate.
All Questions on This Page
45 questions- 1. What does the Spark driver do?
- 2. What is a Spark executor?
- 3. How are jobs, stages, and tasks related?
- 4. What does lazy evaluation mean in Spark?
- 5. What is the difference between a transformation and an action?
- 6. What is a narrow dependency?
- 7. What is a wide dependency?
- 8. Why are Spark shuffles expensive?
- 9. What is Catalyst in Spark SQL?
- 10. What problem does Tungsten solve?
- 11. What is a partition in Spark?
- 12. When should you cache data in Spark?
- 13. What is a broadcast join?
- 14. What is Structured Streaming?
- 15. Why do Spark streaming jobs need checkpoints?
- 16. Walk me through how Spark runs a DataFrame job.
- 17. How do you read a large Parquet table efficiently?
- 18. A Spark job is slow. What do you check first?
- 19. How do you choose a join strategy in Spark?
- 20. How do you handle a skewed join?
- 21. How do you reduce shuffle cost?
- 22. How do you decide partition count?
- 23. How do you decide whether to cache a DataFrame?
- 24. How do you troubleshoot executor memory errors?
- 25. How do you control Spark output file size?
- 26. Design a Spark Structured Streaming pipeline from Kafka to a table.
- 27. How do you handle late events in Structured Streaming?
- 28. What do you look for in a Spark explain plan?
- 29. Why would you avoid a Python UDF?
- 30. How do you run many Spark jobs on a shared cluster?
- 31. A job fails after collect is called. What happened?
- 32. One task takes much longer than the rest. What do you suspect?
- 33. A broadcast join crashes executors. What do you change?
- 34. A Spark job creates thousands of tiny Parquet files. Why is that bad?
- 35. A Spark SQL query scans too much data. What do you inspect?
- 36. A cached DataFrame does not speed up the job. Why?
- 37. A Structured Streaming job keeps falling behind. What do you inspect?
- 38. A streaming job writes duplicate rows after restart. What is missing?
- 39. A groupBy explodes runtime. What is your answer?
- 40. A candidate uses RDDs for a SQL-style pipeline. What concern do you raise?
- 41. A source adds a column and Spark jobs fail. What do you do?
- 42. Executors are lost during heavy stages. What do you check?
- 43. A table is partitioned by user id. Why might this be wrong?
- 44. What can adaptive query execution change at runtime?
- 45. What makes a Spark pipeline production-ready?
Spark Fundamentals
Foundational15 questionsStart here. These are the definitions and first-principle checks that open most rounds.
Q1. What does the Spark driver do?
The driver runs the application, builds the execution plan, requests resources, schedules tasks, and collects results back to the application when needed.
Do not collect large data to the driver. That is a common production failure.
Watch a deeper explanation
Video: Using Apache Spark to analyze open data (Databricks, YouTube)
Q2. What is a Spark executor?
An executor is a worker process that runs tasks, stores cached data, and reports status back to the driver.
Executor sizing affects parallelism, memory pressure, garbage collection, and task failures.
Executor changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q3. How are jobs, stages, and tasks related?
An action creates a job. Spark splits the job into stages at shuffle boundaries, then runs tasks for partitions inside each stage.
This is the core execution model the question expects you to explain.
Job, stage, task changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Spark execution path
A shuffle boundary usually creates a new stage.
Q4. What does lazy evaluation mean in Spark?
Spark records transformations without executing them until an action is called.
Lazy evaluation lets Spark optimize the whole plan before running it.
Lazy evaluation changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
| Answer part | What to say | Evidence to mention |
|---|---|---|
| Definition | Lazy evaluation in one direct sentence. | Official docs or course material |
| Use case | The work where it changes a decision. | Dataset, model, query, dashboard, or pipeline |
| Risk | What breaks when it is misunderstood. | Metric, log, test result, or review note |
Q5. What is the difference between a transformation and an action?
A transformation creates a new logical dataset, while an action triggers execution and returns or writes a result.
Examples: select and filter are transformations. count, collect, and write are actions.
Transformation and action changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Watch a deeper explanation
Video: Fundamentals of Apache Spark and Azure Databricks (Microsoft Developer, YouTube)
Q6. What is a narrow dependency?
A narrow dependency means each output partition depends on a small set of input partitions and usually avoids a shuffle.
map and filter are common examples.
Narrow dependency changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q7. What is a wide dependency?
A wide dependency means output partitions need data from many input partitions, usually causing a shuffle.
groupBy, reduceByKey, joins, and repartition often create wide dependencies.
Wide dependency changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q8. Why are Spark shuffles expensive?
A shuffle redistributes data across executors, causing network transfer, disk spill, serialization, and stage boundaries.
Reducing shuffle size is often the fastest performance win.
Shuffle changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q9. What is Catalyst in Spark SQL?
Catalyst is Spark SQL's optimizer. It analyzes logical plans, applies rules, and creates optimized physical plans.
Mention predicate pushdown, column pruning, join planning, and explain output.
Catalyst optimizer changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q10. What problem does Tungsten solve?
Tungsten improves Spark execution through memory management, binary processing, and code generation.
It is one reason DataFrames can be faster than hand-written RDD logic.
Tungsten changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q11. What is a partition in Spark?
A partition is a slice of data processed by one task.
Partition count affects parallelism, task overhead, shuffle size, and output file count.
Partition changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q12. When should you cache data in Spark?
Cache data when the same expensive intermediate result is reused across multiple actions or branches.
Caching too much data can evict useful blocks or cause memory pressure.
Cache and persist changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q13. What is a broadcast join?
A broadcast join sends a small table to each executor so the large table does not need a full shuffle join.
It works only when the broadcast side fits memory.
Broadcast join changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Watch a deeper explanation
Video: Apache Spark architecture explained (Databricks, YouTube)
Q14. What is Structured Streaming?
Structured Streaming is Spark's streaming API that processes continuous data using DataFrame-style queries.
Interviews focus on triggers, checkpoints, output modes, state, watermarks, and exactly-once claims.
Structured Streaming changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q15. Why do Spark streaming jobs need checkpoints?
Checkpoints store progress and state so streaming jobs can recover after failure without starting from scratch.
Without reliable checkpoints, stateful streaming recovery is unsafe.
Checkpoint changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Spark Practical Interview Questions
Intermediate15 questionsThese questions test whether you can apply the topic to real data, real code, and messy constraints.
Q16. Walk me through how Spark runs a DataFrame job.
Spark builds a logical plan, optimizes it with Catalyst, creates a physical plan, splits work into jobs, stages, and tasks, then runs tasks across executors.
every stage boundary maps to a shuffle or action.
# Interview check: split data, run baseline, inspect metric
from sklearn.metrics import accuracy_score
print(accuracy_score(y_true, y_pred))Q17. How do you read a large Parquet table efficiently?
Select needed columns, filter early, use partition pruning, avoid unnecessary UDFs, and inspect the physical plan.
Column pruning and predicate pushdown are the key points.
read Parquet efficiently changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q18. A Spark job is slow. What do you check first?
Check Spark UI stages, task duration, shuffle read and write, spill, skew, input size, executor failures, and file counts.
measured bottlenecks before changing code comes first.
debug a slow job changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q19. How do you choose a join strategy in Spark?
Check table sizes, join keys, statistics, skew, partitioning, and whether one side can be broadcast.
Broadcast small dimensions. Avoid forcing broadcast when it does not fit memory.
choose join strategy changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q20. How do you handle a skewed join?
Find hot keys, salt large hot keys, split the hot path, broadcast small tables, or enable adaptive skew handling when available.
Skew is visible as a few long tasks in the Spark UI.
handle skewed joins changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q21. How do you reduce shuffle cost?
Filter early, select fewer columns, pre-aggregate, use broadcast joins, avoid unnecessary repartition, and choose sensible partition counts.
The best shuffle is the one you do not create.
reduce shuffle changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q22. How do you decide partition count?
Use data size, cluster parallelism, task duration, shuffle size, and output file target size.
Too few partitions underuse the cluster. Too many create task overhead and small files.
set partitions changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q23. How do you decide whether to cache a DataFrame?
Cache only if the DataFrame is reused and expensive to recompute, then unpersist when it is no longer needed.
Confirm cache actually reduces work in the UI.
cache correctly changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q24. How do you troubleshoot executor memory errors?
Check spill, cached data, wide rows, collect calls, skew, executor sizing, and serialization.
Increasing memory may hide the issue but not fix bad data movement.
fix memory errors changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q25. How do you control Spark output file size?
Set partition count before write, avoid over-partitioning, compact small files, and choose a file format suited to analytics.
Output files often become the next team's query problem.
write output files changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Watch a deeper explanation
Video: Spark Structured Streaming concepts (Databricks, YouTube)
Q26. Design a Spark Structured Streaming pipeline from Kafka to a table.
Read from Kafka, parse and validate events, apply event-time logic, set watermarks if stateful, write to an idempotent sink, and checkpoint to reliable storage.
Mention bad records, schema changes, and restart behavior.
build a streaming pipeline changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q27. How do you handle late events in Structured Streaming?
Use event time, watermarks, state windows, and a business rule for when data is too late to update.
Late data is a product decision as much as a technical one.
handle late events changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q28. What do you look for in a Spark explain plan?
Look for scans, filters, exchanges, joins, broadcast decisions, pushed filters, column pruning, and adaptive execution changes.
Exchange usually means data movement.
use explain changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q29. Why would you avoid a Python UDF?
A Python UDF can block Catalyst optimization and add serialization overhead, so prefer built-in functions when possible.
UDFs are not forbidden, but they need a reason.
replace UDF changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Spark Advanced Scenarios
Advanced15 questionsAdvanced rounds test trade-offs, failure modes, and whether the decision can hold up under production pressure.
Q31. A job fails after collect is called. What happened?
Too much data was pulled to the driver.
Use distributed writes, limit for inspection, or aggregate before collecting.
collect crash changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q32. One task takes much longer than the rest. What do you suspect?
Data skew, large partition, slow node, or uneven input split.
Check task input size, shuffle read, and executor host in the UI.
one slow task changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q33. A broadcast join crashes executors. What do you change?
Do not broadcast that table, reduce the broadcast side, raise threshold only if memory supports it, or use a shuffle join with better partitioning.
Broadcast joins are fast only when the small side is truly small.
Q34. A Spark job creates thousands of tiny Parquet files. Why is that bad?
It slows downstream reads and creates metadata overhead.
Repartition or compact to target file sizes before publishing.
small files after write changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q35. A Spark SQL query scans too much data. What do you inspect?
Check partition filters, pushed filters, selected columns, file format, table stats, and whether UDFs block optimization.
A missing date filter can turn a cheap query into a full scan.
slow Spark SQL query changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q36. A cached DataFrame does not speed up the job. Why?
It may not be reused, may not fit memory, may be evicted, or the expensive step is elsewhere.
Storage tab and stage timing show whether cache helped.
cache no benefit changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q37. A Structured Streaming job keeps falling behind. What do you inspect?
Check input rate, processing rate, trigger time, state size, sink latency, partitions, skew, and checkpoint health.
Increasing cluster size is only one possible fix.
streaming lag changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q38. A streaming job writes duplicate rows after restart. What is missing?
The sink or write logic is not idempotent, or checkpoint handling is wrong.
Exactly-once behavior depends on source, sink, checkpoint, and write pattern.
duplicate streaming output changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q39. A groupBy explodes runtime. What is your answer?
groupBy creates a shuffle. Reduce input, pre-aggregate, fix skew, and choose partitions carefully.
Explain what data moves across the network.
wide transformation changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q40. A candidate uses RDDs for a SQL-style pipeline. What concern do you raise?
They may be bypassing Catalyst optimization and writing more code than needed.
RDDs still fit custom low-level logic, but structured analytics usually fits DataFrames or SQL.
RDD versus DataFrame changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q41. A source adds a column and Spark jobs fail. What do you do?
Check schema enforcement, reader options, downstream selects, table contracts, and compatibility rules.
Additive changes are easy only when code and contracts allow them.
schema drift changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q42. Executors are lost during heavy stages. What do you check?
Check memory overhead, container limits, spill, node health, logs, and shuffle fetch failures.
Executor loss can be infrastructure, memory, or shuffle related.
executor lost changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q43. A table is partitioned by user id. Why might this be wrong?
High-cardinality partitions create too many directories and small files.
Date or event time is often a better top-level partition for analytics.
bad partition key changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q44. What can adaptive query execution change at runtime?
It can adjust join strategy, coalesce shuffle partitions, and handle skew based on runtime statistics.
It helps, but it does not replace good data layout.
adaptive query execution changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
Q45. What makes a Spark pipeline production-ready?
Clear inputs, schema checks, idempotent writes, monitoring, alerting, retries, checkpoints where needed, and documented ownership.
A working notebook is not the same as an operated pipeline.
production readiness changes Spark decisions by affecting the chosen method, the failure mode, or the validation step. Reliable evidence comes from a metric, test result, query output, log, or review note.
RDD vs DataFrame vs Spark SQL
Spark has multiple APIs. the question expects you to know when low-level control is useful and when optimized relational execution is the better choice.
| API | Best use | Optimizer support | Interview warning |
|---|---|---|---|
| RDD | Low-level transformations and custom control | Limited | Verbose and easy to make slow |
| DataFrame | Structured data with columns and schemas | Catalyst and Tungsten | Still needs good partition and join choices |
| Spark SQL | SQL analytics and warehouse-style work | Catalyst and cost-based rules | Query text can hide bad scans or shuffles |
| Dataset | Typed API mainly used in Scala and Java | Catalyst with type safety | Less common in PySpark rounds |
Spark answer signals
execution reasoning more than API recall is the technical point.
- Execution model: driver, executors, jobs, stages, tasks
- Shuffle control: joins, groupBy, repartition, skew
- SQL plans: Catalyst, explain, predicate pushdown
- Operations: Spark UI, retries, memory, checkpoints
How to Prepare for a Spark Interview
Prepare by tracing a Spark application from code to physical execution. A good coverage names the API, shows the plan, identifies the shuffle, and explains the production risk.
- Review driver, executors, cluster manager, jobs, stages, tasks, and partitions.
- lazy evaluation, narrow dependencies, wide dependencies, and shuffles is the explanation path.
- Know Spark SQL basics: Catalyst, predicate pushdown, broadcast joins, file pruning, and explain plans.
- One debugging story involving skew, memory pressure, streaming lag, or small files is useful.
Spark interview prep flow
Most rounds reward clear reasoning more than memorized phrasing.
How Strong Spark Answers Sound
A strong Spark answer starts with the data shape and the execution plan. Say what is being read, how it is partitioned, where the shuffle happens, which join strategy is expected, and how you verified it in the Spark UI or with explain output. production-ready answers also discuss skew, memory pressure, checkpointing, file layout, retries, and cost.
The expensive step
Spark performance problems usually come from shuffles, skew, wide scans, bad joins, tiny files, or memory pressure.
Use the plan as evidence
Explain plans and the Spark UI are better evidence than guessing. Point to stages, task duration, shuffle read, spill, and input size.
Treat cache as a tool, not a habit
Cache repeated expensive work. Do not cache one-time reads or data that does not fit memory.
Separate batch from streaming
Structured Streaming adds state, watermarks, triggers, sinks, and checkpointing. Batch tuning alone is not enough.
Test Yourself: Spark Quiz
Ready to test your Spark knowledge?
6 questions, about 4 minutes. Score 70% or higher to earn a shareable certificate.
Frequently Asked Questions
What do Spark interviews usually ask?
Should I learn RDDs for Spark interviews?
What Spark project should I discuss?
What separates experienced Spark candidates?
Practice Spark answers with coding-style follow-ups
Use this Spark bank for architecture and debugging answers, then practice SQL and Python exercises with Hyring's AI Coding Interviewer before the technical screen.
See Hyring AI Coding InterviewerSources
Written by Adithyan RK