Data Engineering Zoomcamp 2026

Homework 6: Batch Statistics

Distribution of scores and reported study time for this homework.

Submissions

524

Median total score

6

Average total score

6

Score distribution

All values are points.

Questions score

Min
1
Median
6.0
Max
6
Q1
6.0
Avg
5.6
Q3
6.0

Learning in public score

Min
-
Median
0.0
Max
7
Q1
0.0
Avg
0.6
Q3
1.0

Total score

Min
1
Median
6.0
Max
14
Q1
6.0
Avg
6.3
Q3
6.0

Time distribution

All values are hours reported by students.

Lectures

Min
0.0
Median
6.0
Max
180.0
Q1
4.0
Avg
10.2
Q3
10.0

Homework

Min
0.0
Median
2.0
Max
240.0
Q1
1.5
Avg
4.9
Q3
4.0

Question breakdown

Correctness and answer distribution per question.

1. Install Spark and PySpark

524 / 524 correct (100.0%)

Answer Count
4.1.1 206
Spark version: 4.1.1 42
'4.1.1' 40
3.5.0 19
4.0.2 17
pip install pyspark 15
3.5.8 9
3.5.3 7
3.5.5 7
Spark Version: 4.1.1 4
Done 4
Spark version 4.1.1 4
3.5.1 4
Spark version: 3.5.8 3
'3.5.0' 3
4.0.1 3
3.4.0 3
uv add pyspark or pip install pyspark 3
3.5 2
'3.5.8' 2
3.3.2 2
4.0.0 2
yes 2
Spark version: 3.5.1 2
uv add pyspark 2
Spark version: 4.0.1 2
import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder \ .master("local[*]") \ .appName('homework') \ .getOrCreate() print(f"Spark version: {spark.version}") 2
Spark version: 3.5.0 2
Spark version: 4.1.1 +---+ | id| +---+ | 0| | 1| | 2| | 3| | 4| | 5| | 6| | 7| | 8| | 9| +---+ 2
Version: 4.1.1 2
3.5.4 2
Spark 4.4.1 2
!pip install pyspark 2
'3.5.3' 2
version 3.5.0 2
Spark version: 3.3.2 2
Spark Version: 4.1.1 1
v4.1.1 1
done 1
import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder \ .master("local[*]") \ .appName('test') \ .getOrCreate() 1
3.x.x (Check via spark.version) 1
Command used: pip install pyspark Verification: pyspark --version 1
Spark Version : 4.1.1 1
spark version: '4.1.1' 1
3.3.5 1
Make sure you have java 17 installed as Spark requires it. Then download Spark from 1
3.8.4 1
spark = SparkSession.builder \ .master("local[*]") \ .appName("DE_Zoomcamp_Module6") \ .getOrCreate() spark.version 1
Installed Spark and PySpark 1
Spark Version: 4.4.1 1
import os import argparse import pyspark from pyspark.sql import SparkSession from pyspark.sql import functions as F os.environ['SPARK_HOME'] = os.path.expanduser('~/05-batch_processing/spark-3.5.8-bin-hadoop3') 1
Successfully built pyspark Installing collected packages: py4j, pyspark Successfully installed py4j-0.10.9.9 pyspark-4.1.1 1
Spark version: 4.1.1 and pyspark>=4.1.1 1
version 4.1.1 1
pip install pyspark==4.1.1 1
? 1
wget https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-11.parquet -> in data/data_lake Installing Java sudo apt update sudo apt install default-jdk java --version Output (example): openjdk 21.0.10 2026-01-20 OpenJDK Runtime Environment (build 21.0.10+7-Ubuntu-124.04) OpenJDK 64-Bit Server VM (build 21.0.10+7-Ubuntu-124.04, mixed mode, sharing) Set JAVA_HOME (add to your .bashrc or .zshrc): export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java)))) export PATH="${JAVA_HOME}/bin:${PATH}" PySpark recommend using uv for managing Python packages: uv init uv add pyspark ```python import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder \ .master("local[*]") \ .appName("Homework") \ .getOrCreate() spark.version ``` 1
'3.0.3' 1
3.4.4 1
4.1.0 1
yes ,http://localhost:4040/jobs/, spark version -4.1.1 1
PySpark 3.5.0 (Java 17) 1
Yes 1
4.4.1 1
brew install apache-spark, pip install pyspark==3.5.0 1
pip install pyspark==3.x 1
spark.version=4.1.1 1
spar version is 4.1.1 1
'4.0.0' 1
Spark Version: 3.5.3 1
'4.0.1' 1
wget https://www.apache.org/dyn/closer.lua/spark/spark-4.0.2/spark-4.0.2-bin-hadoop3.tgz 1
Version 4.1.1 1
import pyspark from pyspark.sql import SparkSession (version: 4.1.1) 1
pyspark 1
import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder \ .master("local[*]") \ .appName('test') \ .get_builder() print(spark.version) 1
Spark version: 4.0.1 The output confirms that Spark 4.x is successfully installed and running locally. 1
Already had PySpark 4.1.1 installed 1
Spark version 3.5.3 1
YES 1
26/03/09 14:47:39 WARN Utils: Your hostname, Joyes-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 10.0.0.117 instead (on interface en0) 26/03/09 14:47:39 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.5.1 /_/ Using Scala version 2.12.18, OpenJDK 64-Bit Server VM, 11.0.28 Branch HEAD Compiled by user heartsavior on 2024-02-15T11:24:58Z Revision fd86f85e181fc2dc0f50a096855acf83a6cc5d9c Url https://github.com/apache/spark Type --help for more information. The PySpark 3.5.1 version is running... 1
3.5.4 , Had installed previously 1
uv run pip install spark 1
3.5.5 (The course uses Spark 3.5.x; the version installed here via pip install pyspark is 4.1.1) 1
from pyspark.sql import SparkSession spark = SparkSession.builder \ .master("local[*]") \ .appName('test') \ .getOrCreate() print(f"Spark version: {spark.version}") 1
OK. 1
Spark version is 3.5.0 1
WARNING: Using incubator modules: jdk.incubator.vector Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties 26/03/10 14:29:48 WARN Utils: Your hostname, codespaces-a08826, resolves to a loopback address: 127.0.0.1; using 10.0.3.56 instead (on interface eth0) 26/03/10 14:29:48 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 26/03/10 14:29:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 1
pip install pyspark, uv run python test_spark.py 1
PySpark instaled & Spark Sesion created Spark version: 3.5.8 1
spark.version output: 4.1.1 1
from pyspark.sql import SparkSession spark = SparkSession.builder \ .appName("Homework6") \ .master("local[*]") \ .getOrCreate() print(f"Spark version: {spark.version}") 1
PySpark version: 4.1.1. Spark version: 4.1.1 1
spark.version = 3.5.1 1
'3.5.1' 1
Install Spark Run PySpark Create a local spark session Execute spark.version. Spark version: 3.5.3 1
Yes. Java 8. Spark version: 3.5.8 1
Answer: 4.1.1 1
3.5.7 1
The spark version: 3.5.0 1
Spark version: 3.4.1 1
installed 1
Spark version: 3.3.4 1
2.5.0 1
Spark version :- 4.1.0 1
3.5. 1
To verify your installation, you typically run spark.version. For the 2026 Zoomcamp, most students are using Spark 3.5.0 or newer. Ensure your JAVA_HOME and SPARK_HOME are set correctly in your environment 1
3.11.4 1
spark.version '3.5.1' 1
spark.version # Output: '4.1.1' 1
import pyspark 1
brew install openjdk apache-spark && pip install pyspark 1
SparkSession - in-memory SparkContext Spark UI Version v4.0.2 Master local[*] AppName Homework_week6 1
pip install 1
3.4.2 1
Create a local spark session 1
spark.version: 4.1.1 1
uv pip install pyspark and uv run python -c "import pyspark; print(pyspark.__version__)" -> spark version 4.1.1 1
sudo apt install default-jdk -y && pip install pyspark && cd ~/spark && wget https://archive.apache.org/dist/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3.tgz && tar -xzf spark-3.5.1-bin-hadoop3.tgz 1

2. Yellow November 2025

466 / 524 correct (88.9%)

1 6MB 10 (1.9%)
2 25MB 466 (88.9%)
3 75MB 37 (7.1%)
4 100MB 9 (1.7%)

3. Count records

484 / 524 correct (92.4%)

1 62,610 10 (1.9%)
2 102,340 15 (2.9%)
3 162,604 484 (92.4%)
4 225,768 12 (2.3%)

4. Longest trip

462 / 524 correct (88.2%)

1 22.7 3 (0.6%)
2 58.2 7 (1.3%)
3 90.6 462 (88.2%)
4 134.5 47 (9.0%)

5. User Interface

499 / 524 correct (95.2%)

1 80 3 (0.6%)
2 443 7 (1.3%)
3 4040 499 (95.2%)
4 8080 10 (1.9%)

6. Least frequent pickup location zone

498 / 524 correct (95.0%)

1 Governor's Island/Ellis Island/Liberty Island 396 (75.6%)
2 Arden Heights 102 (19.5%)
3 Rikers Island 9 (1.7%)
4 Jamaica Bay 12 (2.3%)

Calculated: 22 March 2026, 09:16