Data Engineering Zoomcamp 2024

Homework 5: Batch processing Statistics

Distribution of scores and reported study time for this homework.

Submissions

446

Median total score

6

Average total score

7

Score distribution

All values are points.

Questions score

Min
1
Median
6.0
Max
6
Q1
6.0
Avg
5.7
Q3
6.0

Learning in public score

Min
-
Median
0.0
Max
7
Q1
0.0
Avg
0.8
Q3
1.0

Total score

Min
2
Median
6.0
Max
14
Q1
6.0
Avg
6.6
Q3
7.0

Time distribution

All values are hours reported by students.

Lectures

Min
1.0
Median
6.0
Max
56.0
Q1
4.0
Avg
9.1
Q3
12.0

Homework

Min
0.0
Median
2.0
Max
40.0
Q1
1.5
Avg
3.3
Q3
4.0

Question breakdown

Correctness and answer distribution per question.

1. Spark version

446 / 446 correct (100.0%)

Answer Count
3.3.2 130
3.5.0 91
3.5.1 85
'3.3.2' 27
3.4.2 19
'3.5.0' 15
'3.5.1' 8
res0: String = 3.3.2 6
3.4.1 5
3.0.3 5
res0: String = 3.5.1 4
'3.4.2' 3
3.3.4 3
v3.3.2 3
version 3.5.0 3
3.3.3 2
PySpark version: 3.3.2 2
res0: String = 3.5.0 2
3.2.1 2
3.4.0 2
Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.5.0 /_/ Using Scala version 2.12.18, OpenJDK 64-Bit Server VM, 17.0.2 Branch HEAD Compiled by user ubuntu on 2023-09-09T01:53:20Z Revision ce5ddad990373636e94071e7cef2f31021add07b Url https://github.com/apache/spark Type --help for more information. 1
res0: String = 3.4.2 1
It shows the version of spark I’m running. Exact output: '3.5.0' 1
Spark version 3.5.0 1
res1: String = 3.3.2 1
3.4.2` 1
Welcome to 3.0.0 1
version 3.4.2 1
res1: String = 3.5.1 1
3.5.1. 1
3 1
Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 24/02/25 20:36:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 24/02/25 20:36:01 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 24/02/25 20:36:16 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(G1 Concurrent GC), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors 1
res0: String = 3.3.4 1
Version v3.5.1 1
Cell In[3], line 1 spark.version. ^ SyntaxError: invalid syntax 1
>>> spark.version '3.4.2' >>> 1
'3.5.0', I installed it before 2-3 months ago, however now in the site there is only '3.5.1'. 1
‘3.3.2’ 1
3.2.0 1
Spark 3.4.2 1
version 3.3.2 1
invalid syntax (spark.version. because this last dot, without this, the version in 3.3.2) 1
String = 3.5.0 1
3.5.0 (the current latest version is 3.5.0 if we installed without specifying a particular version) 1
5.5.1 1
Version v3.1.1 1

2. FHV October 2019 partition size

412 / 446 correct (92.4%)

1 1MB 11 (2.5%)
2 6MB 412 (92.4%)
3 25MB 17 (3.8%)
4 87MB 4 (0.9%)

3. Count records on 15th of October

433 / 446 correct (97.1%)

1 108,164 3 (0.7%)
2 12,856 4 (0.9%)
3 452,470 3 (0.7%)
4 62,610 433 (97.1%)

4. The longest trip

412 / 446 correct (92.4%)

1 631,152.50 Hours 412 (92.4%)
2 243.44 Hours 7 (1.6%)
3 7.68 Hours 12 (2.7%)
4 3.32 Hours 9 (2.0%)

5. Spark UI port

436 / 446 correct (97.8%)

1 80 0 (0.0%)
2 443 3 (0.7%)
3 4040 436 (97.8%)
4 8080 6 (1.3%)

6. Least frequent pickup location zone

421 / 446 correct (94.4%)

1 East Chelsea 8 (1.8%)
2 Jamaica Bay 421 (94.4%)
3 Union Sq 4 (0.9%)
4 Crown Heights North 7 (1.6%)

Calculated: 13 October 2024, 17:46