Data Engineering Zoomcamp 2024

Homework 6: Streaming Statistics

Distribution of scores and reported study time for this homework.

Submissions

199

Median total score

7

Average total score

8

Score distribution

All values are points.

Questions score

Min
5
Median
7.0
Max
7
Q1
6.5
Avg
6.7
Q3
7.0

Learning in public score

Min
-
Median
0.0
Max
7
Q1
0.0
Avg
0.8
Q3
1.0

Total score

Min
5
Median
7.0
Max
15
Q1
7.0
Avg
7.7
Q3
8.0

Time distribution

All values are hours reported by students.

Lectures

Min
0.0
Median
6.0
Max
48.0
Q1
3.0
Avg
8.2
Q3
10.0

Homework

Min
0.5
Median
4.0
Max
42.0
Q1
2.0
Avg
4.8
Q3
5.3

Question breakdown

Correctness and answer distribution per question.

1. Redpanda version

199 / 199 correct (100.0%)

Answer Count
v22.3.5 (rev 28b2443) 147
v22.3.5 22
v23.2.26 (rev 328d83a06e) 8
rpk version v23.2.26 (rev 328d83a06e) 4
22.3.5 3
redpanda:v23.2.26 1
v22.3.5 - 28b2443c33b0c2f8807bc5c460cc2478cfee1044-dirty 1
rpk version v23.3.6 (rev 78642d6969) 1
v23.3.11 1
redpanda: v23.3.11 1
answer: v22.3.5 (rev 28b2443) 1
redpanda-admin version 23.3 1
docker.redpanda.com/vectorized/redpanda:v22.3.5 1
redpanda@80a3958407e6:/$ rpk version v22.3.5 (rev 28b2443) 1
v23.2.26 - 328d83a06e743eaa53773b1246c260bdfd3e88b1 1
rpk version = 23.3 1
Version: v23.3.8 Git ref: 25f1bd3294 Build date: 2024-03-16T04:56:21Z OS/Arch: linux/amd64 Go version: go1.21.3 1

2. Creating a topic

199 / 199 correct (100.0%)

Answer Count
TOPIC STATUS test-topic OK 112
TOPIC STATUS test-topic OK 11
test-topic OK 11
rpk topic create test-topic 5
rpk topic create test-topic TOPIC STATUS test-topic OK 5
docker exec redpanda-1 rpk topic create test-topic 2
test-topic 2
sudo docker exec redpanda-1 rpk topic create test-topic TOPIC STATUS test-topic OK 2
TOPIC STATUS test-topic OK 2
TOPIC test-topic STATUS OK 2
TOPIC test-topic STATUS OK 2
TOPIC STATUS\n test-topic OK 1
TOPIC STATUS test-topic OK 1
TOPIC STATUS test-topic OK 1
$ rpk topic create 'test-topic' TOPIC STATUS test-topic OK 1
green_tripdata_2020-10_csv 1
TOPIC STATUS test-topic OK 1
$ docker exec -it redpanda-1 rpk topic create test-topic TOPIC STATUS test-topic OK 1
kafka-topics.sh --bootstrap-server localhost:9092 --topic topic_name --create 1
ritikatilwalia@Ritikas-MBP homework % docker exec -it c2c874738fd7 rpk topic create test-topic TOPIC STATUS test-topic OK 1
TOPIC STATUS test-topic OK` 1
Created topic test-topic 1
{ "topic": "mi-topic", "partitions": 1, "replicas": 1, "leader": 0 } 1
redpanda@863be22baea8:/$ rpk topic create test-topic TOPIC STATUS test-topic OK 1
TOPIC STATUS test-topik OK 1
topic: test-topic status: ok 1
Created topic "test-topic". 1
TOPIC | STATUS test-topic | OK 1
Sent: {'number': 0} Sent: {'number': 1} Sent: {'number': 2} Sent: {'number': 3} Sent: {'number': 4} Sent: {'number': 5} Sent: {'number': 6} Sent: {'number': 7} Sent: {'number': 8} Sent: {'number': 9} t1 took 0.62 seconds t2 took 0.00 seconds 1
redpanda@9f1284af9d59:/$ rpk topic create test-topic TOPIC STATUS test-topic OK 1
TOPIC: hallo-topic, STATUS: OK 1
TOPIC test-topic STATUS OK 1
redpanda@80a3958407e6:/$ rpk topic create test-topic TOPIC STATUS test-topic OK redpanda@80a3958407e6:/$ 1
TOPIC STATUS my-topic OK 1
TOPIC STATUS test-topic OK 1
rpk topic create test-topic/ TOPIC STATUS/ test-topic OK/ 1
$ winpty docker exec -it redpanda-1 rpk topic create test-topic TOPIC STATUS test-topic OK 1
redpanda@3a2839133b9b:/$ rpk topic create test-topic | TOPIC "test-topic" | STATUS "OK" 1
test-topic TOPIC STATUS test-topic OK 1
docker exec -it redpanda-1 rpk topic create test-topic 1
$ rpk topic create test-topic TOPIC STATUS test-topic OK 1
TOPIC STATUS test-topic OK 1
docker exec -it redpanda-1 bash -c "rpk topic create test-topic" 1
NAME PARTITIONS REPLICAS test-topic 1 1 1
test-topic OK 1
TOPIC STATUS test-topic OK 1
redpanda@775bed156a4a:/$ rpk topic create test-topic TOPIC STATUS test-topic OK 1
! docker exec -it redpanda-1 rpk topic create test-topic 1
$ rpk topic create test-topic TOPIC STATUS test-topic OK 1
docker exec -it 238d0eb4b1d1 rpk topic create 'test-topic' 1

3. Connecting to the Kafka server

192 / 199 correct (96.5%)

1 True 192 (96.5%)
2 False 0 (0.0%)
3 {"connected": True} 3 (1.5%)
4 {"status": "SUCCESS"} 3 (1.5%)

4. Sending data to the stream

151 / 199 correct (75.9%)

1 Sending the messages 151 (75.9%)
2 Flushing 8 (4.0%)
3 Both took approximately the same amount of time 39 (19.6%)

5. Time to send data

199 / 199 correct (100.0%)

Answer Count
64 4
84.72 seconds 3
18 3
47 3
53 3
24 3
16 2
50 2
40 2
32 2
49 seconds 2
23 2
20 2
102 2
63 seconds 2
142 2
0.52 seconds 2
33 2
44 seconds 2
0.51 seconds 2
91 sec 2
51 2
79.08 seconds 2
51.26 1
took 92.48 seconds 1
took 28.09 seconds 1
31 seconds 1
0.66 seconds 1
5 minute 1
total time: 48.31 seconds 1
took 58.23 seconds 1
63.79 seconds 1
52s 1
23 seconds 1
10 minutes 1
21 1
Elapsed time: 49.756105184555054 seconds 1
1 second 1
37 1
56.38 1
1.63 1
44.06 seconds 1
took 52.54 seconds 1
77 1
241 1
65 1
59.07 seconds 1
Sending messages to topic: green-trips Total messages sent: 476386.00 rows Sending the messages time took: 61.96 seconds Flushing time took: 0.04 seconds 1
0.51 1
between 45-48 seconds 1
15 1
32s 1
20.79996 1
30.82 seconds 1
245.96 seconds 1
22 1
47.41 seconds 1
took 188 seconds 1
3 1
377.77 seconds 1
716.72 second 1
39.45 seconds 1
took 115.52 seconds 1
60 1
17 1
took 173.73 seconds to send data 1
37 sec 1
51 seconds 1
IOPub data rate exceeded. The Jupyter server will temporarily stop sending output to the client in order to avoid crashing it. 1
61 1
took 18.16 seconds 1
40.29 1
took 68.44 seconds 1
104 sec 1
48 1
82 1
188secs 1
82.25 seconds 1
57 milliseconds 1
66 seconds 1
DONE in 84 seconds 1
12.794867992401123 seconds 1
170 1
73 seconds 1
82 seconds 1
Time taken to send data: 94 seconds 1
233.14 seconds 1
35 1
131 seconds 1
134 1
34.37242889404297 1
84 seconds 1
It depends on various factors 1
53 seconds 1
Time taken to send data: 65 seconds 1
50.44 seconds 1
14 seconds 1
36.12 seconds 1
took 46.76 seconds 1
took 52.00 seconds 1
0.5099530220031738 1
25 1
61 seconds 1
78 seconds 1
49 sec 1
148.41 seconds 1
124 1
55.4 1
64.59 seconds (about 1 minute) 1
26 seconds 1
2 1
0.01 seconds 1
75 1
36 1
45 1
7.32 seconds 1
It took 129 seconds to send the data 1
68 1
45 seconds 1
25s 1
Execution time (seconds): 33 1
1.29 1
107.03 seconds 1
0.37 1
35 minutes 1
18 seconds 1
24.78 seconds 1
253 1
took 0.87 seconds 1
25 seconds 1
39.27 seconds 1
79.03 1
67.47 seconds 1
18.38 seconds 1
67 1
97 seconds 1
53 sec 1
106 seconds 1
77 seconds 1
176 1
47s 1
took 6.78 seconds 1
97 1
Took 176.89 seconds 1
Took about 63 seconds 1
43.89seconds 1
47.06 seconds (running on a Google VM with a rate=10128.46it/s) 1
112 1
147.95 seconds 1
67.31 seconds 1
24 minutes 1
41 1
Script execution completed in 66.40 seconds. 1
143 1
97.84 1
58 seconds 1
45s 1
0.54second 1
51.13 seconds 1
22.4 seconds 1
145 1
63.1 1
148 1
253.41 1
1 min 1
148 seconds 1
83.88 seconds 1

6. Parsing the data

199 / 199 correct (100.0%)

Answer Count
Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0) 76
DataFrame[lpep_pickup_datetime: string, lpep_dropoff_datetime: string, PULocationID: int, DOLocationID: int, passenger_count: double, trip_distance: double, tip_amount: double] 24
root |-- lpep_pickup_datetime: string (nullable = true) |-- lpep_dropoff_datetime: string (nullable = true) |-- PULocationID: integer (nullable = true) |-- DOLocationID: integer (nullable = true) |-- passenger_count: double (nullable = true) |-- trip_distance: double (nullable = true) |-- tip_amount: double (nullable = true) 4
Row(lpep_pickup_datetime='2019-10-04 17:57:52', lpep_dropoff_datetime='2019-10-04 18:10:24', PULocationID=185, DOLocationID=51, passenger_count=1.0, trip_distance=3.28, tip_amount=0.0) 3
+--------------------+---------------------+------------+------------+---------------+-------------+----------+ |lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ | 2019-10-01 00:26:02| 2019-10-01 00:39:58| 112| 196| 1.0| 5.88| 0.0| | 2019-10-01 00:18:11| 2019-10-01 00:22:38| 43| 263| 1.0| 0.8| 0.0| | 2019-10-01 00:09:31| 2019-10-01 00:24:47| 255| 228| 2.0| 7.5| 0.0| | 2019-10-01 00:37:40| 2019-10-01 00:41:49| 181| 181| 1.0| 0.9| 0.0| | 2019-10-01 00:08:13| 2019-10-01 00:17:56| 97| 188| 1.0| 2.52| 2.26| | 2019-10-01 00:35:01| 2019-10-01 00:43:40| 65| 49| 1.0| 1.47| 1.86| | 2019-10-01 00:28:09| 2019-10-01 00:30:49| 7| 179| 1.0| 0.6| 1.0| | 2019-10-01 00:28:26| 2019-10-01 00:32:01| 41| 74| 1.0| 0.56| 0.0| | 2019-10-01 00:14:01| 2019-10-01 00:26:16| 255| 49| 1.0| 2.42| 0.0| | 2019-10-01 00:03:03| 2019-10-01 00:17:13| 130| 131| 1.0| 3.4| 2.85| | 2019-10-01 00:07:10| 2019-10-01 00:23:38| 24| 74| 3.0| 3.18| 0.0| | 2019-10-01 00:25:48| 2019-10-01 00:49:52| 255| 188| 1.0| 4.7| 1.0| | 2019-10-01 00:03:12| 2019-10-01 00:14:43| 129| 160| 1.0| 3.1| 0.0| | 2019-10-01 00:44:56| 2019-10-01 00:51:06| 18| 169| 1.0| 1.19| 0.25| | 2019-10-01 00:55:14| 2019-10-01 01:00:49| 223| 7| 1.0| 1.09| 1.46| | 2019-10-01 00:06:06| 2019-10-01 00:11:05| 75| 262| 1.0| 1.24| 2.01| | 2019-10-01 00:00:19| 2019-10-01 00:14:32| 97| 228| 1.0| 3.03| 3.58| | 2019-10-01 00:09:31| 2019-10-01 00:20:41| 41| 74| 1.0| 2.03| 2.16| | 2019-10-01 00:30:36| 2019-10-01 00:34:30| 41| 42| 1.0| 0.73| 1.26| | 2019-10-01 00:58:32| 2019-10-01 01:05:08| 41| 116| 1.0| 1.48| 0.0| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ 3
Row(lpep_pickup_datetime='2019-10-01 00:57:44', lpep_dropoff_datetime='2019-10-01 01:06:59', PULocationID=25, DOLocationID=228, passenger_count=1.0, trip_distance=2.62, tip_amount=0.0) 3
Row(lpep_pickup_datetime=None, lpep_dropoff_datetime=None, PULocationID=None, DOLocationID=None, passenger_count=None, trip_distance=None, tip_amount=None) 3
| 2019-10-01 00:26:02| 2019-10-01 00:39:58| 112| 196| 1.0| 5.88| 0.0| 2
+--------------------+---------------------+------------+------------+---------------+-------------+----------+ |lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ | 2019-10-01 00:26:02| 2019-10-01 00:39:58| 112| 196| 1| 5.88| 0.0| | 2019-10-01 00:18:11| 2019-10-01 00:22:38| 43| 263| 1| 0.8| 0.0| | 2019-10-01 00:09:31| 2019-10-01 00:24:47| 255| 228| 2| 7.5| 0.0| | 2019-10-01 00:37:40| 2019-10-01 00:41:49| 181| 181| 1| 0.9| 0.0| | 2019-10-01 00:08:13| 2019-10-01 00:17:56| 97| 188| 1| 2.52| 2.26| | 2019-10-01 00:35:01| 2019-10-01 00:43:40| 65| 49| 1| 1.47| 1.86| | 2019-10-01 00:28:09| 2019-10-01 00:30:49| 7| 179| 1| 0.6| 1.0| | 2019-10-01 00:28:26| 2019-10-01 00:32:01| 41| 74| 1| 0.56| 0.0| | 2019-10-01 00:14:01| 2019-10-01 00:26:16| 255| 49| 1| 2.42| 0.0| | 2019-10-01 00:03:03| 2019-10-01 00:17:13| 130| 131| 1| 3.4| 2.85| | 2019-10-01 00:07:10| 2019-10-01 00:23:38| 24| 74| 3| 3.18| 0.0| | 2019-10-01 00:25:48| 2019-10-01 00:49:52| 255| 188| 1| 4.7| 1.0| | 2019-10-01 00:03:12| 2019-10-01 00:14:43| 129| 160| 1| 3.1| 0.0| | 2019-10-01 00:44:56| 2019-10-01 00:51:06| 18| 169| 1| 1.19| 0.25| | 2019-10-01 00:55:14| 2019-10-01 01:00:49| 223| 7| 1| 1.09| 1.46| | 2019-10-01 00:06:06| 2019-10-01 00:11:05| 75| 262| 1| 1.24| 2.01| | 2019-10-01 00:00:19| 2019-10-01 00:14:32| 97| 228| 1| 3.03| 3.58| | 2019-10-01 00:09:31| 2019-10-01 00:20:41| 41| 74| 1| 2.03| 2.16| | 2019-10-01 00:30:36| 2019-10-01 00:34:30| 41| 42| 1| 0.73| 1.26| | 2019-10-01 00:58:32| 2019-10-01 01:05:08| 41| 116| 1| 1.48| 0.0| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ 2
root |-- lpep_pickup_datetime: string (nullable = true) |-- lpep_dropoff_datetime: string (nullable = true) |-- PULocationID: integer (nullable = true) |-- DOLocationID: integer (nullable = true) |-- passenger_count: double (nullable = true) |-- trip_distance: double (nullable = true) |-- tip_amount: double (nullable = true) |-- timestamp: timestamp (nullable = true) 2
Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0): string, lpep_dropoff_datetime: string, PULocationID: int, DOLocationID: int, passenger_count: double, trip_distance: double, tip_amount: double] 2
Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=None, trip_distance=5.0, tip_amount=0.0) 1
lpep_pickup_datetime='2019-10-01 00:57:44', lpep_dropoff_datetime='2019-10-01 01:06:59', PULocationID=25, DOLocationID=228, passenger_count=1.0, trip_distance=2.62, tip_amount=0.0 1
root |-- key: binary (nullable = true) |-- value: binary (nullable = true) |-- topic: string (nullable = true) |-- partition: integer (nullable = true) |-- offset: long (nullable = true) |-- timestamp: timestamp (nullable = true) |-- timestampType: integer (nullable = true) 1
def streamData(): DataFrame = { val kafkaDF = spark.readStream.format("kafka") .option("kafka.bootstrap.servers", "server:port") .option("subscribe", "topic_name") .load() val df:DataFrame = kafkaDF.select(col("topic"), expr("cast (value as String) as data")) df.select(from_json(col("data"), schema).as("data")) .selectExpr("data.hive_schema.tablename") .writeStream .format("console") .option("truncate", value = false) .outputMode("append") .start() .awaitTermination() } 1
Row(lpep_pickup_datetime='2019-10-01T00:26:02', lpep_dropoff_datetime='2019-10-01T00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0) 1
|{"lpep_pickup_datetime": "2019-10-01 00:26:02", "lpep_dropoff_datetime": "2019-10-01 00:39:58", "PULocationID": 112, "DOLocationID": 196, "passenger_count": 1.0, "trip_distance": 5.88, "tip_amount": 0.0}| 1
Row(lpep_pickup_datetime='2019-10-01 01:12:49', lpep_dropoff_datetime='2019-10-01 01:12:54', PULocationID=242, DOLocationID=242, passenger_count=2.0, trip_distance=0.0, tip_amount=4.2) 1
Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=None, DOLocationID=None, passenger_count=None, trip_distance=None, tip_amount=None) 1
Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:26:02", "lpep_dropoff_datetime": "2019-10-01 00:39:58", "PULocationID": 112, "DOLocationID": 196, "passenger_count": 1.0, "trip_distance": 5.88, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=0, timestamp=datetime.datetime(2024, 4, 7, 15, 47, 38, 655000), timestampType=0) 1
Row(lpep_pickup_datetime='2019-10-01 01:53:53', lpep_dropoff_datetime='2019-10-01 02:36:09', PULocationID=181, DOLocationID=216, passenger_count=1.0, trip_distance=8.66, tip_amount=0.0) 1
------------------------------------------- Batch: 0 ------------------------------------------- +--------------------+---------------------+------------+------------+---------------+-------------+----------+ |lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ | 2019-10-01 00:26:02| 2019-10-01 00:39:58| 112| 196| 1.0| 5.88| 0.0| | 2019-10-01 00:18:11| 2019-10-01 00:22:38| 43| 263| 1.0| 0.8| 0.0| | 2019-10-01 00:09:31| 2019-10-01 00:24:47| 255| 228| 2.0| 7.5| 0.0| | 2019-10-01 00:37:40| 2019-10-01 00:41:49| 181| 181| 1.0| 0.9| 0.0| | 2019-10-01 00:08:13| 2019-10-01 00:17:56| 97| 188| 1.0| 2.52| 2.26| | 2019-10-01 00:35:01| 2019-10-01 00:43:40| 65| 49| 1.0| 1.47| 1.86| 1
Row(lpep_pickup_datetime='2019-10-01 01:11:13', lpep_dropoff_datetime='2019-10-01 01:20:35', PULocationID=256, DOLocationID=225, passenger_count=1.0, trip_distance=1.88, tip_amount=1.96) 1
2019-10-01 00:26:02 |2019-10-01 00:39:58 |112 |196 |1.0 |5.88 |0.0 1
Batch ID: 2316 +-----------------------+-----------------------+------------+------------+---------------+-------------+----------+ |lpep_pickup_datetime |lpep_dropoff_datetime |PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +-----------------------+-----------------------+------------+------------+---------------+-------------+----------+ |["2019-10-24 10:42:00"]|["2019-10-24 10:42:00"]|25 |25 |null |null |null | |["2019-10-24 10:59:00"]|["2019-10-24 11:52:00"]|101 |164 |null |null |null | |["2019-10-24 10:20:00"]|["2019-10-24 11:06:00"]|92 |237 |null |null |null | |["2019-10-24 10:14:00"]|["2019-10-24 10:28:00"]|32 |60 |null |null |null | |["2019-10-24 10:49:00"]|["2019-10-24 11:11:00"]|108 |154 |null |null |null | |["2019-10-24 10:30:00"]|["2019-10-24 10:38:00"]|21 |21 |null |null |null | |["2019-10-24 10:43:00"]|["2019-10-24 10:53:00"]|155 |155 |null |null |null | |["2019-10-24 10:06:00"]|["2019-10-24 10:17:00"]|72 |39 |null |null |null | |["2019-10-24 10:20:00"]|["2019-10-24 10:52:00"]|188 |26 |null |null |null | |["2019-10-24 10:31:00"]|["2019-10-24 10:57:00"]|52 |88 |null |null |null | |["2019-10-24 10:43:00"]|["2019-10-24 10:56:00"]|130 |216 |null |null |null | |["2019-10-24 10:28:00"]|["2019-10-24 10:39:00"]|64 |131 |null |null |null | |["2019-10-24 10:33:00"]|["2019-10-24 10:51:00"]|39 |188 |null |null |null | |["2019-10-24 10:38:00"]|["2019-10-24 11:23:00"]|86 |108 |null |null |null | |["2019-10-24 10:41:00"]|["2019-10-24 11:02:00"]|3 |51 |null |null |null | |["2019-10-24 10:51:00"]|["2019-10-24 11:04:00"]|197 |10 |null |null |null | |["2019-10-24 10:55:00"]|["2019-10-24 11:56:00"]|55 |231 |null |null |null | |["2019-10-24 10:57:00"]|["2019-10-24 11:10:00"]|263 |74 |null |null |null | |["2019-10-24 10:26:00"]|["2019-10-24 10:49:00"]|155 |35 |null |null |null | |["2019-10-24 10:59:00"]|["2019-10-24 11:11:00"]|35 |188 |null |null |null | +-----------------------+-----------------------+------------+------------+---------------+-------------+----------+ only showing top 20 rows 1
DataFrame[lpep_pickup_datetime: string, lpep_dropoff_datetime: string, PULocationID: int, DOLocationID: int, passenger_count: double, trip_distance: double, tip_amount: double, timestamp: timestamp] 1
[Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0)] 1
Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58',PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0) 1
|-- lpep_pickup_datetime: string (nullable = true) |-- lpep_dropoff_datetime: string (nullable = true) |-- PULocationID: integer (nullable = true) |-- DOLocationID: integer (nullable = true) |-- passenger_count: double (nullable = true) |-- trip_distance: double (nullable = true) |-- tip_amount: double (nullable = true) 1
2019-10-01 00:26:02 |2019-10-01 00:39:58 |112 |196 |1.0 |5.88 |0.0 | +----------------------+---------------------+------------+-------------+---------------+-------------+----------+ 1
# +--------------------+---------------------+------------+------------+---------------+-------------+----------+ # |lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| # +--------------------+---------------------+------------+------------+---------------+-------------+----------+ # | 2019-10-01 00:26:02| 2019-10-01 00:39:58| 112| 196| 1.0| 5.88| 0.0| # | 2019-10-01 00:18:11| 2019-10-01 00:22:38| 43| 263| 1.0| 0.8| 0.0| # | 2019-10-01 00:09:31| 2019-10-01 00:24:47| 255| 228| 2.0| 7.5| 0.0| # | 2019-10-01 00:37:40| 2019-10-01 00:41:49| 181| 181| 1.0| 0.9| 0.0| # | 2019-10-01 00:08:13| 2019-10-01 00:17:56| 97| 188| 1.0| 2.52| 2.26| # | 2019-10-01 00:35:01| 2019-10-01 00:43:40| 65| 49| 1.0| 1.47| 1.86| # | 2019-10-01 00:28:09| 2019-10-01 00:30:49| 7| 179| 1.0| 0.6| 1.0| # | 2019-10-01 00:28:26| 2019-10-01 00:32:01| 41| 74| 1.0| 0.56| 0.0| # | 2019-10-01 00:14:01| 2019-10-01 00:26:16| 255| 49| 1.0| 2.42| 0.0| # | 2019-10-01 00:03:03| 2019-10-01 00:17:13| 130| 131| 1.0| 3.4| 2.85| # | 2019-10-01 00:07:10| 2019-10-01 00:23:38| 24| 74| 3.0| 3.18| 0.0| # | 2019-10-01 00:25:48| 2019-10-01 00:49:52| 255| 188| 1.0| 4.7| 1.0| # | 2019-10-01 00:03:12| 2019-10-01 00:14:43| 129| 160| 1.0| 3.1| 0.0| # | 2019-10-01 00:44:56| 2019-10-01 00:51:06| 18| 169| 1.0| 1.19| 0.25| # | 2019-10-01 00:55:14| 2019-10-01 01:00:49| 223| 7| 1.0| 1.09| 1.46| # | 2019-10-01 00:06:06| 2019-10-01 00:11:05| 75| 262| 1.0| 1.24| 2.01| # | 2019-10-01 00:00:19| 2019-10-01 00:14:32| 97| 228| 1.0| 3.03| 3.58| # | 2019-10-01 00:09:31| 2019-10-01 00:20:41| 41| 74| 1.0| 2.03| 2.16| # | 2019-10-01 00:30:36| 2019-10-01 00:34:30| 41| 42| 1.0| 0.73| 1.26| # | 2019-10-01 00:58:32| 2019-10-01 01:05:08| 41| 116| 1.0| 1.48| 0.0| # +--------------------+---------------------+------------+------------+---------------+-------------+----------+ 1
<pyspark.sql.streaming.StreamingQuery at 0x7f86dd7718a0> Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0) 1
+------------+------------+---------------------+--------------------+---------------+----------+-------------+ |DOLocationID|PULocationID|lpep_dropoff_datetime|lpep_pickup_datetime|passenger_count|tip_amount|trip_distance| +------------+------------+---------------------+--------------------+---------------+----------+-------------+ | 196| 112| 2019-10-01 00:39:58| 2019-10-01 00:26:02| 1.0| 0.0| 5.88| | 263| 43| 2019-10-01 00:22:38| 2019-10-01 00:18:11| 1.0| 0.0| 0.8| | 228| 255| 2019-10-01 00:24:47| 2019-10-01 00:09:31| 2.0| 0.0| 7.5| | 181| 181| 2019-10-01 00:41:49| 2019-10-01 00:37:40| 1.0| 0.0| 0.9| | 188| 97| 2019-10-01 00:17:56| 2019-10-01 00:08:13| 1.0| 2.26| 2.52| | 49| 65| 2019-10-01 00:43:40| 2019-10-01 00:35:01| 1.0| 1.86| 1.47| | 179| 7| 2019-10-01 00:30:49| 2019-10-01 00:28:09| 1.0| 1.0| 0.6| | 74| 41| 2019-10-01 00:32:01| 2019-10-01 00:28:26| 1.0| 0.0| 0.56| | 49| 255| 2019-10-01 00:26:16| 2019-10-01 00:14:01| 1.0| 0.0| 2.42| | 131| 130| 2019-10-01 00:17:13| 2019-10-01 00:03:03| 1.0| 2.85| 3.4| +------------+------------+---------------------+--------------------+---------------+----------+-------------+ 1
Row(VendorID=None, lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0) 1
|lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---|2019-10-01 00:26:02 |2019-10-01 00:39:58 |112 |196 |1.0 |5.88 |0.0 | 1
+--------------------+---------------------+------------+------------+---------------+-------------+----------+ |lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ | 2019-10-01 00:26:02| 2019-10-01 00:39:58| 112| 196| 1.0| 5.88| 0.0| | 2019-10-01 00:18:11| 2019-10-01 00:22:38| 43| 263| 1.0| 0.8| 0.0| | 2019-10-01 00:09:31| 2019-10-01 00:24:47| 255| 228| 2.0| 7.5| 0.0| | 2019-10-01 00:37:40| 2019-10-01 00:41:49| 181| 181| 1.0| 0.9| 0.0| | 2019-10-01 00:08:13| 2019-10-01 00:17:56| 97| 188| 1.0| 2.52| 2.26| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ 1
Deserializing the data 1
lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ | 2019-10-01 00:00:48| 2019-10-01 00:05:42| 210| 108| 2.0| 1.03| 2.19| 1
lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ | 2019-10-01 00:00:48| 2019-10-01 00:05:42| 210| 108| 2.0| 1.03| 2.19| | 2019-10-01 00:45:08| 2019-10-01 01:04:28| 83| 36| 2.0| 5.75| 0.0| | 2019-10-01 00:32:44| 2019-10-01 00:46:53| 92| 260| 2.0| 6.01| 4.16| | 2019-10-01 00:05:09| 2019-10-01 00:18:34| 75| 119| 1.0| 1.08| 2.36| | 2019-10-01 00:18:16| 2019-10-01 00:53:16| 29| 92| 1.0| 21.39| 0.0| | 2019-10-01 00:03:02| 2019-10-01 00:09:54| 129| 129| 2.0| 2.23| 0.0| | 2019-10-01 00:48:33| 2019-10-01 00:53:42| 74| 168| 1.0| 1.24| 2.34| | 2019-10-01 00:03:09| 2019-10-01 00:28:01| 82| 75| 1.0| 8.19| 0.0| | 2019-10-01 00:02:26| 2019-10-01 00:12:24| 74| 69| 2.0| 2.28| 2.0| | 2019-10-01 00:25:15| 2019-10-01 00:47:25| 129| 28| 1.0| 9.22| 0.0| | 2019-10-01 00:44:27| 2019-10-01 00:47:17| 223| 223| 1.0| 0.86| 1.16| | 2019-10-01 00:08:52| 2019-10-01 00:59:25| 61| 18| 1.0| 18.82| 0.0| | 2019-10-01 00:39:21| 2019-10-01 00:44:12| 95| 95| 1.0| 1.03| 0.0| | 2019-10-01 00:17:46| 2019-10-01 00:28:49| 130| 205| 1.0| 3.69| 2.76| | 2019-10-01 01:02:12| 2019-10-01 01:06:15| 130| 197| 1.0| 1.21| 1.82| | 2019-10-01 00:24:10| 2019-10-01 00:42:01| 130| 145| 1.0| 10.34| 6.16| | 2019-10-01 00:17:49| 2019-10-01 00:33:39| 7| 157| 3.0| 3.1| 0.0| | 2019-10-01 00:46:07| 2019-10-01 00:51:22| 129| 129| 1.0| 1.1| 1.7| | 2019-10-01 00:08:33| 2019-10-01 00:22:10| 66| 79| 2.0| 5.08| 0.0| | 2019-10-01 00:47:09| 2019-10-01 00:49:58| 82| 129| 1.0| 0.6| 0.0| 1
{"lpep_pickup_datetime": "2019-10-01 00:26:02", "lpep_dropoff_datetime": "2019-10-01 00:39:58", "PULocationID": 112, "DOLocationID": 196, "passenger_count": 1.0, "trip_distance": 5.88, "tip_amount": 0.0} 1
2019-10-01 00:26:02 |2019-10-01 00:39:58 |112 |196 |1.0 |5.88 |0.0 | 1
------------------------------------------- Batch: 0 ------------------------------------------- +--------------------+---------------------+------------+------------+---------------+-------------+----------+ |lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ | 2019-10-01 00:26:02| 2019-10-01 00:39:58| 112| 196| 1.0| 5.88| 0.0| | 2019-10-01 00:18:11| 2019-10-01 00:22:38| 43| 263| 1.0| 0.8| 0.0| | 2019-10-01 00:09:31| 2019-10-01 00:24:47| 255| 228| 2.0| 7.5| 0.0| | 2019-10-01 00:37:40| 2019-10-01 00:41:49| 181| 181| 1.0| 0.9| 0.0| | 2019-10-01 00:08:13| 2019-10-01 00:17:56| 97| 188| 1.0| 2.52| 2.26| | 2019-10-01 00:35:01| 2019-10-01 00:43:40| 65| 49| 1.0| 1.47| 1.86| | 2019-10-01 00:28:09| 2019-10-01 00:30:49| 7| 179| 1.0| 0.6| 1.0| | 2019-10-01 00:28:26| 2019-10-01 00:32:01| 41| 74| 1.0| 0.56| 0.0| | 2019-10-01 00:14:01| 2019-10-01 00:26:16| 255| 49| 1.0| 2.42| 0.0| | 2019-10-01 00:03:03| 2019-10-01 00:17:13| 130| 131| 1.0| 3.4| 2.85| | 2019-10-01 00:07:10| 2019-10-01 00:23:38| 24| 74| 3.0| 3.18| 0.0| | 2019-10-01 00:25:48| 2019-10-01 00:49:52| 255| 188| 1.0| 4.7| 1.0| | 2019-10-01 00:03:12| 2019-10-01 00:14:43| 129| 160| 1.0| 3.1| 0.0| | 2019-10-01 00:44:56| 2019-10-01 00:51:06| 18| 169| 1.0| 1.19| 0.25| | 2019-10-01 00:55:14| 2019-10-01 01:00:49| 223| 7| 1.0| 1.09| 1.46| | 2019-10-01 00:06:06| 2019-10-01 00:11:05| 75| 262| 1.0| 1.24| 2.01| | 2019-10-01 00:00:19| 2019-10-01 00:14:32| 97| 228| 1.0| 3.03| 3.58| | 2019-10-01 00:09:31| 2019-10-01 00:20:41| 41| 74| 1.0| 2.03| 2.16| | 2019-10-01 00:30:36| 2019-10-01 00:34:30| 41| 42| 1.0| 0.73| 1.26| | 2019-10-01 00:58:32| 2019-10-01 01:05:08| 41| 116| 1.0| 1.48| 0.0| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ only showing top 20 rows 1
------------------------------------------- Batch: 0 ------------------------------------------- +--------------------+---------------------+------------+------------+---------------+-------------+----------+ |lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ |2019-10-01 01:41:39 |2019-10-01 01:45:06 |82 |173 |2.0 |0.72 |0.0 | |2019-10-01 01:16:30 |2019-10-01 01:22:42 |82 |83 |1.0 |1.27 |1.56 | |2019-10-01 01:03:24 |2019-10-01 01:19:34 |75 |244 |1.0 |4.05 |0.0 | |2019-10-01 01:26:12 |2019-10-01 01:34:34 |260 |157 |2.0 |1.43 |0.0 | |2019-10-01 01:25:27 |2019-10-01 01:41:16 |168 |250 |1.0 |6.16 |4.16 | |2019-10-01 01:19:41 |2019-10-01 01:31:19 |7 |226 |5.0 |2.52 |0.0 | |2019-10-01 01:00:56 |2019-10-01 01:19:12 |82 |179 |1.0 |4.94 |0.0 | |2019-10-01 01:22:03 |2019-10-02 01:01:18 |188 |61 |5.0 |2.78 |0.0 | |2019-10-01 06:58:06 |2019-10-01 07:18:48 |74 |74 |1.0 |0.48 |0.0 | |2019-10-01 07:23:01 |2019-10-01 07:38:03 |74 |75 |1.0 |0.61 |0.0 | |2019-10-01 01:02:21 |2019-10-01 01:09:46 |82 |129 |1.0 |1.01 |1.56 | |2019-10-01 01:23:02 |2019-10-01 01:30:37 |129 |157 |1.0 |1.44 |0.0 | |2019-10-01 01:22:48 |2019-10-01 01:28:32 |260 |226 |1.0 |0.88 |0.0 | |2019-10-01 01:36:51 |2019-10-01 01:44:32 |260 |7 |1.0 |1.7 |2.64 | |2019-10-01 01:11:00 |2019-10-01 02:14:45 |244 |187 |1.0 |28.21 |0.0 | |2019-10-01 01:26:23 |2019-10-01 01:41:24 |80 |233 |1.0 |5.52 |0.0 | |2019-10-01 01:03:32 |2019-10-01 01:05:56 |75 |74 |1.0 |1.04 |0.0 | |2019-10-01 01:28:34 |2019-10-01 01:42:09 |75 |47 |1.0 |4.17 |0.0 | |2019-10-01 01:38:26 |2019-10-01 01:47:35 |179 |223 |1.0 |1.07 |0.0 | |2019-10-01 01:27:08 |2019-10-01 01:43:26 |256 |198 |1.0 |4.13 |0.0 | +--------------------+---------------------+------------+------------+---------------+-------------+----------+ only showing top 20 rows 1
<bound method DataFrame.show of DataFrame[lpep_pickup_datetime: string, lpep_dropoff_datetime: string, PULocationID: int, DOLocationID: int, passenger_count: double, trip_distance: double, tip_amount: double]> 1
|lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ | 2019-10-01 00:26:02| 2019-10-01 00:39:58| 112| 196| 1| 5.88| 0.0| 1
Row(lpep_pickup_datetime='2019-10-01 09:05:07', lpep_dropoff_datetime='2019-10-01 09:11:30', PULocationID=74, DOLocationID=166, passenger_count=1.0, trip_distance=1.4, tip_amount=0.0) 1
Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:26:02", "lpep_dropoff_datetime": "2019-10-01 00:39:58", "PULocationID": 112, "DOLocationID": 196, "passenger_count": 1.0, "trip_distance": 5.88, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=0, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 35, 270000), timestampType=0)Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:26:02", "lpep_dropoff_datetime": "2019-10-01 00:39:58", "PULocationID": 112, "DOLocationID": 196, "passenger_count": 1.0, "trip_distance": 5.88, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=0, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 35, 270000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:26:02", "lpep_dropoff_datetime": "2019-10-01 00:39:58", "PULocationID": 112, "DOLocationID": 196, "passenger_count": 1.0, "trip_distance": 5.88, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=1, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 35, 322000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:26:02", "lpep_dropoff_datetime": "2019-10-01 00:39:58", "PULocationID": 112, "DOLocationID": 196, "passenger_count": 1.0, "trip_distance": 5.88, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=1, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 35, 322000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:08:13", "lpep_dropoff_datetime": "2019-10-01 00:17:56", "PULocationID": 97, "DOLocationID": 188, "passenger_count": 1.0, "trip_distance": 2.52, "tip_amount": 2.26}'), topic='green-trips', partition=0, offset=45, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 37, 582000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:08:13", "lpep_dropoff_datetime": "2019-10-01 00:17:56", "PULocationID": 97, "DOLocationID": 188, "passenger_count": 1.0, "trip_distance": 2.52, "tip_amount": 2.26}'), topic='green-trips', partition=0, offset=44, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 37, 531000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:08:13', lpep_dropoff_datetime='2019-10-01 00:17:56', PULocationID=97, DOLocationID=188, passenger_count=1.0, trip_distance=2.52, tip_amount=2.26) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:35:01", "lpep_dropoff_datetime": "2019-10-01 00:43:40", "PULocationID": 65, "DOLocationID": 49, "passenger_count": 1.0, "trip_distance": 1.47, "tip_amount": 1.86}'), topic='green-trips', partition=0, offset=55, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 38, 103000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:35:01", "lpep_dropoff_datetime": "2019-10-01 00:43:40", "PULocationID": 65, "DOLocationID": 49, "passenger_count": 1.0, "trip_distance": 1.47, "tip_amount": 1.86}'), topic='green-trips', partition=0, offset=56, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 38, 155000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:35:01', lpep_dropoff_datetime='2019-10-01 00:43:40', PULocationID=65, DOLocationID=49, passenger_count=1.0, trip_distance=1.47, tip_amount=1.86) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:28:09", "lpep_dropoff_datetime": "2019-10-01 00:30:49", "PULocationID": 7, "DOLocationID": 179, "passenger_count": 1.0, "trip_distance": 0.6, "tip_amount": 1.0}'), topic='green-trips', partition=0, offset=66, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 38, 672000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:28:09", "lpep_dropoff_datetime": "2019-10-01 00:30:49", "PULocationID": 7, "DOLocationID": 179, "passenger_count": 1.0, "trip_distance": 0.6, "tip_amount": 1.0}'), topic='green-trips', partition=0, offset=66, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 38, 672000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:28:26', lpep_dropoff_datetime='2019-10-01 00:32:01', PULocationID=41, DOLocationID=74, passenger_count=1.0, trip_distance=0.56, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:28:26", "lpep_dropoff_datetime": "2019-10-01 00:32:01", "PULocationID": 41, "DOLocationID": 74, "passenger_count": 1.0, "trip_distance": 0.56, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=75, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 39, 136000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:28:26", "lpep_dropoff_datetime": "2019-10-01 00:32:01", "PULocationID": 41, "DOLocationID": 74, "passenger_count": 1.0, "trip_distance": 0.56, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=76, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 39, 187000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:14:01', lpep_dropoff_datetime='2019-10-01 00:26:16', PULocationID=255, DOLocationID=49, passenger_count=1.0, trip_distance=2.42, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:14:01", "lpep_dropoff_datetime": "2019-10-01 00:26:16", "PULocationID": 255, "DOLocationID": 49, "passenger_count": 1.0, "trip_distance": 2.42, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=84, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 39, 598000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:14:01", "lpep_dropoff_datetime": "2019-10-01 00:26:16", "PULocationID": 255, "DOLocationID": 49, "passenger_count": 1.0, "trip_distance": 2.42, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=85, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 39, 650000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:03:03', lpep_dropoff_datetime='2019-10-01 00:17:13', PULocationID=130, DOLocationID=131, passenger_count=1.0, trip_distance=3.4, tip_amount=2.85) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:03:03", "lpep_dropoff_datetime": "2019-10-01 00:17:13", "PULocationID": 130, "DOLocationID": 131, "passenger_count": 1.0, "trip_distance": 3.4, "tip_amount": 2.85}'), topic='green-trips', partition=0, offset=93, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 40, 62000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:03:03", "lpep_dropoff_datetime": "2019-10-01 00:17:13", "PULocationID": 130, "DOLocationID": 131, "passenger_count": 1.0, "trip_distance": 3.4, "tip_amount": 2.85}'), topic='green-trips', partition=0, offset=95, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 40, 164000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:07:10", "lpep_dropoff_datetime": "2019-10-01 00:23:38", "PULocationID": 24, "DOLocationID": 74, "passenger_count": 3.0, "trip_distance": 3.18, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=102, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 40, 526000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:07:10', lpep_dropoff_datetime='2019-10-01 00:23:38', PULocationID=24, DOLocationID=74, passenger_count=3.0, trip_distance=3.18, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:07:10", "lpep_dropoff_datetime": "2019-10-01 00:23:38", "PULocationID": 24, "DOLocationID": 74, "passenger_count": 3.0, "trip_distance": 3.18, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=104, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 40, 628000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:25:48", "lpep_dropoff_datetime": "2019-10-01 00:49:52", "PULocationID": 255, "DOLocationID": 188, "passenger_count": 1.0, "trip_distance": 4.7, "tip_amount": 1.0}'), topic='green-trips', partition=0, offset=111, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 40, 988000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:25:48', lpep_dropoff_datetime='2019-10-01 00:49:52', PULocationID=255, DOLocationID=188, passenger_count=1.0, trip_distance=4.7, tip_amount=1.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:25:48", "lpep_dropoff_datetime": "2019-10-01 00:49:52", "PULocationID": 255, "DOLocationID": 188, "passenger_count": 1.0, "trip_distance": 4.7, "tip_amount": 1.0}'), topic='green-trips', partition=0, offset=113, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 41, 89000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:25:48", "lpep_dropoff_datetime": "2019-10-01 00:49:52", "PULocationID": 255, "DOLocationID": 188, "passenger_count": 1.0, "trip_distance": 4.7, "tip_amount": 1.0}'), topic='green-trips', partition=0, offset=119, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 41, 398000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:03:12', lpep_dropoff_datetime='2019-10-01 00:14:43', PULocationID=129, DOLocationID=160, passenger_count=1.0, trip_distance=3.1, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:03:12", "lpep_dropoff_datetime": "2019-10-01 00:14:43", "PULocationID": 129, "DOLocationID": 160, "passenger_count": 1.0, "trip_distance": 3.1, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=124, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 41, 657000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:03:12", "lpep_dropoff_datetime": "2019-10-01 00:14:43", "PULocationID": 129, "DOLocationID": 160, "passenger_count": 1.0, "trip_distance": 3.1, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=128, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 41, 865000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:44:56', lpep_dropoff_datetime='2019-10-01 00:51:06', PULocationID=18, DOLocationID=169, passenger_count=1.0, trip_distance=1.19, tip_amount=0.25) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:44:56", "lpep_dropoff_datetime": "2019-10-01 00:51:06", "PULocationID": 18, "DOLocationID": 169, "passenger_count": 1.0, "trip_distance": 1.19, "tip_amount": 0.25}'), topic='green-trips', partition=0, offset=136, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 42, 279000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:44:56", "lpep_dropoff_datetime": "2019-10-01 00:51:06", "PULocationID": 18, "DOLocationID": 169, "passenger_count": 1.0, "trip_distance": 1.19, "tip_amount": 0.25}'), topic='green-trips', partition=0, offset=139, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 42, 432000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:55:14', lpep_dropoff_datetime='2019-10-01 01:00:49', PULocationID=223, DOLocationID=7, passenger_count=1.0, trip_distance=1.09, tip_amount=1.46) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:55:14", "lpep_dropoff_datetime": "2019-10-01 01:00:49", "PULocationID": 223, "DOLocationID": 7, "passenger_count": 1.0, "trip_distance": 1.09, "tip_amount": 1.46}'), topic='green-trips', partition=0, offset=146, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 42, 793000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:55:14", "lpep_dropoff_datetime": "2019-10-01 01:00:49", "PULocationID": 223, "DOLocationID": 7, "passenger_count": 1.0, "trip_distance": 1.09, "tip_amount": 1.46}'), topic='green-trips', partition=0, offset=148, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 42, 895000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:55:14', lpep_dropoff_datetime='2019-10-01 01:00:49', PULocationID=223, DOLocationID=7, passenger_count=1.0, trip_distance=1.09, tip_amount=1.46) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:06:06", "lpep_dropoff_datetime": "2019-10-01 00:11:05", "PULocationID": 75, "DOLocationID": 262, "passenger_count": 1.0, "trip_distance": 1.24, "tip_amount": 2.01}'), topic='green-trips', partition=0, offset=154, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 43, 206000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:06:06", "lpep_dropoff_datetime": "2019-10-01 00:11:05", "PULocationID": 75, "DOLocationID": 262, "passenger_count": 1.0, "trip_distance": 1.24, "tip_amount": 2.01}'), topic='green-trips', partition=0, offset=157, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 43, 361000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:06:06', lpep_dropoff_datetime='2019-10-01 00:11:05', PULocationID=75, DOLocationID=262, passenger_count=1.0, trip_distance=1.24, tip_amount=2.01) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:00:19", "lpep_dropoff_datetime": "2019-10-01 00:14:32", "PULocationID": 97, "DOLocationID": 228, "passenger_count": 1.0, "trip_distance": 3.03, "tip_amount": 3.58}'), topic='green-trips', partition=0, offset=163, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 43, 672000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:00:19", "lpep_dropoff_datetime": "2019-10-01 00:14:32", "PULocationID": 97, "DOLocationID": 228, "passenger_count": 1.0, "trip_distance": 3.03, "tip_amount": 3.58}'), topic='green-trips', partition=0, offset=165, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 43, 776000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:00:19', lpep_dropoff_datetime='2019-10-01 00:14:32', PULocationID=97, DOLocationID=228, passenger_count=1.0, trip_distance=3.03, tip_amount=3.58) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:09:31", "lpep_dropoff_datetime": "2019-10-01 00:20:41", "PULocationID": 41, "DOLocationID": 74, "passenger_count": 1.0, "trip_distance": 2.03, "tip_amount": 2.16}'), topic='green-trips', partition=0, offset=171, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 44, 85000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:09:31", "lpep_dropoff_datetime": "2019-10-01 00:20:41", "PULocationID": 41, "DOLocationID": 74, "passenger_count": 1.0, "trip_distance": 2.03, "tip_amount": 2.16}'), topic='green-trips', partition=0, offset=174, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 44, 238000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:09:31', lpep_dropoff_datetime='2019-10-01 00:20:41', PULocationID=41, DOLocationID=74, passenger_count=1.0, trip_distance=2.03, tip_amount=2.16) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:09:31", "lpep_dropoff_datetime": "2019-10-01 00:20:41", "PULocationID": 41, "DOLocationID": 74, "passenger_count": 1.0, "trip_distance": 2.03, "tip_amount": 2.16}'), topic='green-trips', partition=0, offset=179, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 44, 495000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:30:36", "lpep_dropoff_datetime": "2019-10-01 00:34:30", "PULocationID": 41, "DOLocationID": 42, "passenger_count": 1.0, "trip_distance": 0.73, "tip_amount": 1.26}'), topic='green-trips', partition=0, offset=183, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 44, 701000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:30:36', lpep_dropoff_datetime='2019-10-01 00:34:30', PULocationID=41, DOLocationID=42, passenger_count=1.0, trip_distance=0.73, tip_amount=1.26) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:30:36", "lpep_dropoff_datetime": "2019-10-01 00:34:30", "PULocationID": 41, "DOLocationID": 42, "passenger_count": 1.0, "trip_distance": 0.73, "tip_amount": 1.26}'), topic='green-trips', partition=0, offset=185, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 44, 805000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:58:32", "lpep_dropoff_datetime": "2019-10-01 01:05:08", "PULocationID": 41, "DOLocationID": 116, "passenger_count": 1.0, "trip_distance": 1.48, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=194, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 45, 267000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:58:32', lpep_dropoff_datetime='2019-10-01 01:05:08', PULocationID=41, DOLocationID=116, passenger_count=1.0, trip_distance=1.48, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:58:32", "lpep_dropoff_datetime": "2019-10-01 01:05:08", "PULocationID": 41, "DOLocationID": 116, "passenger_count": 1.0, "trip_distance": 1.48, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=192, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 45, 164000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:11:58", "lpep_dropoff_datetime": "2019-10-01 00:16:21", "PULocationID": 134, "DOLocationID": 134, "passenger_count": 2.0, "trip_distance": 0.89, "tip_amount": 1.36}'), topic='green-trips', partition=0, offset=203, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 45, 728000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:11:58', lpep_dropoff_datetime='2019-10-01 00:16:21', PULocationID=134, DOLocationID=134, passenger_count=2.0, trip_distance=0.89, tip_amount=1.36) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:11:58", "lpep_dropoff_datetime": "2019-10-01 00:16:21", "PULocationID": 134, "DOLocationID": 134, "passenger_count": 2.0, "trip_distance": 0.89, "tip_amount": 1.36}'), topic='green-trips', partition=0, offset=203, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 45, 728000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:42:27", "lpep_dropoff_datetime": "2019-10-01 01:04:30", "PULocationID": 260, "DOLocationID": 92, "passenger_count": 1.0, "trip_distance": 4.21, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=210, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 46, 89000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:42:27', lpep_dropoff_datetime='2019-10-01 01:04:30', PULocationID=260, DOLocationID=92, passenger_count=1.0, trip_distance=4.21, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:42:27", "lpep_dropoff_datetime": "2019-10-01 01:04:30", "PULocationID": 260, "DOLocationID": 92, "passenger_count": 1.0, "trip_distance": 4.21, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=210, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 46, 89000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:42:27', lpep_dropoff_datetime='2019-10-01 01:04:30', PULocationID=260, DOLocationID=92, passenger_count=1.0, trip_distance=4.21, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:42:27", "lpep_dropoff_datetime": "2019-10-01 01:04:30", "PULocationID": 260, "DOLocationID": 92, "passenger_count": 1.0, "trip_distance": 4.21, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=217, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 46, 448000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:42:27", "lpep_dropoff_datetime": "2019-10-01 01:04:30", "PULocationID": 260, "DOLocationID": 92, "passenger_count": 1.0, "trip_distance": 4.21, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=218, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 46, 499000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:03:38", "lpep_dropoff_datetime": "2019-10-01 00:12:43", "PULocationID": 83, "DOLocationID": 56, "passenger_count": 1.0, "trip_distance": 1.64, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=224, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 46, 807000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:03:38', lpep_dropoff_datetime='2019-10-01 00:12:43', PULocationID=83, DOLocationID=56, passenger_count=1.0, trip_distance=1.64, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:03:38", "lpep_dropoff_datetime": "2019-10-01 00:12:43", "PULocationID": 83, "DOLocationID": 56, "passenger_count": 1.0, "trip_distance": 1.64, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=225, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 46, 858000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:12:30", "lpep_dropoff_datetime": "2019-10-01 00:27:16", "PULocationID": 181, "DOLocationID": 89, "passenger_count": 1.0, "trip_distance": 3.6, "tip_amount": 0.74}'), topic='green-trips', partition=0, offset=232, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 47, 217000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:12:30', lpep_dropoff_datetime='2019-10-01 00:27:16', PULocationID=181, DOLocationID=89, passenger_count=1.0, trip_distance=3.6, tip_amount=0.74) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:12:30", "lpep_dropoff_datetime": "2019-10-01 00:27:16", "PULocationID": 181, "DOLocationID": 89, "passenger_count": 1.0, "trip_distance": 3.6, "tip_amount": 0.74}'), topic='green-trips', partition=0, offset=232, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 47, 217000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:12:30", "lpep_dropoff_datetime": "2019-10-01 00:27:16", "PULocationID": 181, "DOLocationID": 89, "passenger_count": 1.0, "trip_distance": 3.6, "tip_amount": 0.74}'), topic='green-trips', partition=0, offset=239, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 47, 574000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:12:30', lpep_dropoff_datetime='2019-10-01 00:27:16', PULocationID=181, DOLocationID=89, passenger_count=1.0, trip_distance=3.6, tip_amount=0.74) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:44:27", "lpep_dropoff_datetime": "2019-10-01 00:54:27", "PULocationID": 181, "DOLocationID": 188, "passenger_count": 1.0, "trip_distance": 2.44, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=240, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 47, 626000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:44:27", "lpep_dropoff_datetime": "2019-10-01 00:54:27", "PULocationID": 181, "DOLocationID": 188, "passenger_count": 1.0, "trip_distance": 2.44, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=246, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 47, 935000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:44:27', lpep_dropoff_datetime='2019-10-01 00:54:27', PULocationID=181, DOLocationID=188, passenger_count=1.0, trip_distance=2.44, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:44:27", "lpep_dropoff_datetime": "2019-10-01 00:54:27", "PULocationID": 181, "DOLocationID": 188, "passenger_count": 1.0, "trip_distance": 2.44, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=249, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 48, 89000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:04:40", "lpep_dropoff_datetime": "2019-10-01 00:10:02", "PULocationID": 226, "DOLocationID": 226, "passenger_count": 1.0, "trip_distance": 1.58, "tip_amount": 1.56}'), topic='green-trips', partition=0, offset=255, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 48, 396000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:04:40', lpep_dropoff_datetime='2019-10-01 00:10:02', PULocationID=226, DOLocationID=226, passenger_count=1.0, trip_distance=1.58, tip_amount=1.56) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:04:40", "lpep_dropoff_datetime": "2019-10-01 00:10:02", "PULocationID": 226, "DOLocationID": 226, "passenger_count": 1.0, "trip_distance": 1.58, "tip_amount": 1.56}'), topic='green-trips', partition=0, offset=256, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 48, 448000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:22:41", "lpep_dropoff_datetime": "2019-10-01 00:31:53", "PULocationID": 33, "DOLocationID": 80, "passenger_count": 1.0, "trip_distance": 4.46, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=263, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 48, 809000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:22:41', lpep_dropoff_datetime='2019-10-01 00:31:53', PULocationID=33, DOLocationID=80, passenger_count=1.0, trip_distance=4.46, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:22:41", "lpep_dropoff_datetime": "2019-10-01 00:31:53", "PULocationID": 33, "DOLocationID": 80, "passenger_count": 1.0, "trip_distance": 4.46, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=264, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 48, 860000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:38:00", "lpep_dropoff_datetime": "2019-10-01 00:45:50", "PULocationID": 256, "DOLocationID": 37, "passenger_count": 1.0, "trip_distance": 1.81, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=270, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 49, 167000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:38:00', lpep_dropoff_datetime='2019-10-01 00:45:50', PULocationID=256, DOLocationID=37, passenger_count=1.0, trip_distance=1.81, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:38:00", "lpep_dropoff_datetime": "2019-10-01 00:45:50", "PULocationID": 256, "DOLocationID": 37, "passenger_count": 1.0, "trip_distance": 1.81, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=272, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 49, 270000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:38:00", "lpep_dropoff_datetime": "2019-10-01 00:45:50", "PULocationID": 256, "DOLocationID": 37, "passenger_count": 1.0, "trip_distance": 1.81, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=278, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 49, 577000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:38:00', lpep_dropoff_datetime='2019-10-01 00:45:50', PULocationID=256, DOLocationID=37, passenger_count=1.0, trip_distance=1.81, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:38:00", "lpep_dropoff_datetime": "2019-10-01 00:45:50", "PULocationID": 256, "DOLocationID": 37, "passenger_count": 1.0, "trip_distance": 1.81, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=279, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 49, 629000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:54:35", "lpep_dropoff_datetime": "2019-10-01 01:01:39", "PULocationID": 80, "DOLocationID": 148, "passenger_count": 1.0, "trip_distance": 2.33, "tip_amount": 1.96}'), topic='green-trips', partition=0, offset=284, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 49, 887000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:54:35', lpep_dropoff_datetime='2019-10-01 01:01:39', PULocationID=80, DOLocationID=148, passenger_count=1.0, trip_distance=2.33, tip_amount=1.96) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:54:35", "lpep_dropoff_datetime": "2019-10-01 01:01:39", "PULocationID": 80, "DOLocationID": 148, "passenger_count": 1.0, "trip_distance": 2.33, "tip_amount": 1.96}'), topic='green-trips', partition=0, offset=287, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 50, 43000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:55:48", "lpep_dropoff_datetime": "2019-10-01 01:00:45", "PULocationID": 223, "DOLocationID": 7, "passenger_count": 5.0, "trip_distance": 0.82, "tip_amount": 1.36}'), topic='green-trips', partition=0, offset=292, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 50, 299000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:55:48', lpep_dropoff_datetime='2019-10-01 01:00:45', PULocationID=223, DOLocationID=7, passenger_count=5.0, trip_distance=0.82, tip_amount=1.36) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:55:48", "lpep_dropoff_datetime": "2019-10-01 01:00:45", "PULocationID": 223, "DOLocationID": 7, "passenger_count": 5.0, "trip_distance": 0.82, "tip_amount": 1.36}'), topic='green-trips', partition=0, offset=295, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 50, 452000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:52:36", "lpep_dropoff_datetime": "2019-10-01 00:52:42", "PULocationID": 92, "DOLocationID": 92, "passenger_count": 1.0, "trip_distance": 0.0, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=300, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 50, 709000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:52:36', lpep_dropoff_datetime='2019-10-01 00:52:42', PULocationID=92, DOLocationID=92, passenger_count=1.0, trip_distance=0.0, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:52:36", "lpep_dropoff_datetime": "2019-10-01 00:52:42", "PULocationID": 92, "DOLocationID": 92, "passenger_count": 1.0, "trip_distance": 0.0, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=302, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 50, 812000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:52:36", "lpep_dropoff_datetime": "2019-10-01 00:52:42", "PULocationID": 92, "DOLocationID": 92, "passenger_count": 1.0, "trip_distance": 0.0, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=307, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 51, 67000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:52:36', lpep_dropoff_datetime='2019-10-01 00:52:42', PULocationID=92, DOLocationID=92, passenger_count=1.0, trip_distance=0.0, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:20:56", "lpep_dropoff_datetime": "2019-10-01 00:27:52", "PULocationID": 129, "DOLocationID": 83, "passenger_count": 1.0, "trip_distance": 1.49, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=311, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 51, 273000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:20:56", "lpep_dropoff_datetime": "2019-10-01 00:27:52", "PULocationID": 129, "DOLocationID": 83, "passenger_count": 1.0, "trip_distance": 1.49, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=315, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 51, 479000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:20:56', lpep_dropoff_datetime='2019-10-01 00:27:52', PULocationID=129, DOLocationID=83, passenger_count=1.0, trip_distance=1.49, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:20:56", "lpep_dropoff_datetime": "2019-10-01 00:27:52", "PULocationID": 129, "DOLocationID": 83, "passenger_count": 1.0, "trip_distance": 1.49, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=318, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 51, 633000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:37:54", "lpep_dropoff_datetime": "2019-10-01 00:50:51", "PULocationID": 130, "DOLocationID": 205, "passenger_count": 1.0, "trip_distance": 3.6, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=323, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 51, 891000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:37:54', lpep_dropoff_datetime='2019-10-01 00:50:51', PULocationID=130, DOLocationID=205, passenger_count=1.0, trip_distance=3.6, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:37:54", "lpep_dropoff_datetime": "2019-10-01 00:50:51", "PULocationID": 130, "DOLocationID": 205, "passenger_count": 1.0, "trip_distance": 3.6, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=326, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 52, 45000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:42:34", "lpep_dropoff_datetime": "2019-10-01 00:45:23", "PULocationID": 179, "DOLocationID": 223, "passenger_count": 1.0, "trip_distance": 0.53, "tip_amount": 1.06}'), topic='green-trips', partition=0, offset=331, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 52, 301000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:42:34", "lpep_dropoff_datetime": "2019-10-01 00:45:23", "PULocationID": 179, "DOLocationID": 223, "passenger_count": 1.0, "trip_distance": 0.53, "tip_amount": 1.06}'), topic='green-trips', partition=0, offset=333, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 52, 402000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:42:34', lpep_dropoff_datetime='2019-10-01 00:45:23', PULocationID=179, DOLocationID=223, passenger_count=1.0, trip_distance=0.53, tip_amount=1.06) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:42:34", "lpep_dropoff_datetime": "2019-10-01 00:45:23", "PULocationID": 179, "DOLocationID": 223, "passenger_count": 1.0, "trip_distance": 0.53, "tip_amount": 1.06}'), topic='green-trips', partition=0, offset=338, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 52, 657000), timestampType=0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:57:00", "lpep_dropoff_datetime": "2019-10-01 00:59:27", "PULocationID": 7, "DOLocationID": 7, "passenger_count": 1.0, "trip_distance": 0.53, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=341, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 52, 812000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:57:00', lpep_dropoff_datetime='2019-10-01 00:59:27', PULocationID=7, DOLocationID=7, passenger_count=1.0, trip_distance=0.53, tip_amount=0.0) Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:57:00", "lpep_dropoff_datetime": "2019-10-01 00:59:27", "PULocationID": 7, "DOLocationID": 7, "passenger_count": 1.0, "trip_distance": 0.53, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=346, timestamp=datetime.datetime(2024, 4, 8, 11, 53, 53, 71000), timestampType=0) Row(lpep_pickup_datetime='2019-10-01 00:57:00', lpep_dropoff_datetime='2019-10-01 00:59:27', PULocationID=7, DOLocationID=7, passenger_count=1.0, trip_distance=0.53, tip_amount=0.0) 1
+--------------------+---------------------+------------+------------+---------------+-------------+----------+ |lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ |2019-10-01 00:26:02 |2019-10-01 00:39:58 |112 |196 |1.0 |5.88 |0.0 | |2019-10-01 00:18:11 |2019-10-01 00:22:38 |43 |263 |1.0 |0.8 |0.0 | |2019-10-01 00:09:31 |2019-10-01 00:24:47 |255 |228 |2.0 |7.5 |0.0 | |2019-10-01 00:37:40 |2019-10-01 00:41:49 |181 |181 |1.0 |0.9 |0.0 | |2019-10-01 00:08:13 |2019-10-01 00:17:56 |97 |188 |1.0 |2.52 |2.26 | |2019-10-01 00:35:01 |2019-10-01 00:43:40 |65 |49 |1.0 |1.47 |1.86 | |2019-10-01 00:28:09 |2019-10-01 00:30:49 |7 |179 |1.0 |0.6 |1.0 | |2019-10-01 00:28:26 |2019-10-01 00:32:01 |41 |74 |1.0 |0.56 |0.0 | |2019-10-01 00:14:01 |2019-10-01 00:26:16 |255 |49 |1.0 |2.42 |0.0 | |2019-10-01 00:03:03 |2019-10-01 00:17:13 |130 |131 |1.0 |3.4 |2.85 | |2019-10-01 00:07:10 |2019-10-01 00:23:38 |24 |74 |3.0 |3.18 |0.0 | 1
Row(lpep_pickup_datetime='2019-10-01 00:09:31', lpep_dropoff_datetime='2019-10-01 00:20:41', PULocationID=41, DOLocationID=74, passenger_count=1.0, trip_distance=2.03, tip_amount=2.16) 1
Row(pickup_datetime='2019-10-01 00:26:02', dropoff_datetime='2019-10-01 00:39:58', pu_location_id=112, do_location_id=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0) 1
lpep_pickup_datetime = 2019-10-01 00:26:02 | lpep_dropoff_datetime = 2019-10-01 00:39:58 | PULocationID = 112 | DOLocationID = 196 | passenger_count = 1.0 | trip_distance = 5.88 | tip_amount = 0.0 1
{ "lpep_pickup_datetime": "2019-10-01 00:26:02", "lpep_dropoff_datetime": "2019-10-01 00:39:58", "PULocationID": 112, "DOLocationID": 196, "passenger_count": 1.0, "trip_distance": 5.88, "tip_amount": 0.0 } 1
|{"lpep_pickup_datetime": ["2019-10-02 16:03:33"], "lpep_dropoff_datetime": ["2019-10-02 16:16:44"], "PULocationID": 166, "DOLocationID": 239, "passenger_count": "1", "trip_distance": 2.31, "total_amount": 18.06}| 1
lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ | 2019-10-01 00:26:02| 2019-10-01 00:39:58| 112| 196| 1.0| 5.88| 0.0| 1
Row(lpep_pickup_datetime='2019-10-01 01:32:57', lpep_dropoff_datetime='2019-10-01 01:47:32', PULocationID=166, DOLocationID=166, passenger_count=1.0, trip_distance=1.95, tip_amount=0.0) 1
+--------------------+---------------------+------------+------------+---------------+-------------+----------+ |lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ | 2019-10-01 00:18:11| 2019-10-01 00:22:38| 43| 263| 1.0| 0.8| 0.0| | 2019-10-01 00:09:31| 2019-10-01 00:24:47| 255| 228| 2.0| 7.5| 0.0| | 2019-10-01 00:37:40| 2019-10-01 00:41:49| 181| 181| 1.0| 0.9| 0.0| | 2019-10-01 00:08:13| 2019-10-01 00:17:56| 97| 188| 1.0| 2.52| 2.26| | 2019-10-01 00:35:01| 2019-10-01 00:43:40| 65| 49| 1.0| 1.47| 1.86| | 2019-10-01 00:28:09| 2019-10-01 00:30:49| 7| 179| 1.0| 0.6| 1.0| | 2019-10-01 00:28:26| 2019-10-01 00:32:01| 41| 74| 1.0| 0.56| 0.0| | 2019-10-01 00:14:01| 2019-10-01 00:26:16| 255| 49| 1.0| 2.42| 0.0| | 2019-10-01 00:03:03| 2019-10-01 00:17:13| 130| 131| 1.0| 3.4| 2.85| | 2019-10-01 00:26:02| 2019-10-01 00:39:58| 112| 196| 1.0| 5.88| 0.0| | 2019-10-01 00:18:11| 2019-10-01 00:22:38| 43| 263| 1.0| 0.8| 0.0| | 2019-10-01 00:09:31| 2019-10-01 00:24:47| 255| 228| 2.0| 7.5| 0.0| | 2019-10-01 00:37:40| 2019-10-01 00:41:49| 181| 181| 1.0| 0.9| 0.0| | 2019-10-01 00:08:13| 2019-10-01 00:17:56| 97| 188| 1.0| 2.52| 2.26| | 2019-10-01 00:35:01| 2019-10-01 00:43:40| 65| 49| 1.0| 1.47| 1.86| | 2019-10-01 00:28:09| 2019-10-01 00:30:49| 7| 179| 1.0| 0.6| 1.0| | 2019-10-01 00:28:26| 2019-10-01 00:32:01| 41| 74| 1.0| 0.56| 0.0| | 2019-10-01 00:14:01| 2019-10-01 00:26:16| 255| 49| 1.0| 2.42| 0.0| | 2019-10-01 00:03:03| 2019-10-01 00:17:13| 130| 131| 1.0| 3.4| 2.85| +--------------------+---------------------+------------+------------+---------------+-------------+----------+ 1
|lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID |DOLocationID |passenger_count|trip_distance|tip_amount | 1
Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, trip_distance=5.880000114440918, tip_amount=0.0) 1
1 1
24/03/25 06:41:28 WARN ResolveWriteToStream: Temporary checkpoint location created which is deleted normally when the query didn't fail: /tmp/temporary-faa0a9fe-1b47-4708-8e99-62f896822edc. If it's required to delete it under any circumstances, please set spark.sql.streaming.forceDeleteTempCheckpointLocation to true. Important to know deleting temp checkpoint folder is best effort. 24/03/25 06:41:28 WARN ResolveWriteToStream: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled. 24/03/25 06:41:28 WARN AdminClientConfig: These configurations '[key.deserializer, value.deserializer, enable.auto.commit, max.poll.records, auto.offset.reset]' were supplied but are not used yet. Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0) 1
|lpep_pickup_datetime|lpep_dropoff_datetime|PULocationID|DOLocationID|passenger_count|trip_distance|tip_amount| 1
Row(lpep_pickup_datetime='2019-10-01 01:41:39', lpep_dropoff_datetime='2019-10-01 01:45:06', PULocationID=82, DOLocationID=173, passenger_count=2.0, trip_distance=0.72, tip_amount=0.0, timestamp=datetime.datetime(2024, 3, 24, 21, 15, 45, 261000)) 1
DataFrame[lpep_pickup_datetime: string, lpep_dropoff_datetime: string, PULocationID: int, DOLocationID: int, passenger_count: double, trip_distance: double, tip_amount: double]Row(lpep_pickup_datetime='2019-10-01 00:44:56', lpep_dropoff_datetime='2019-10-01 00:51:06', PULocationID=18, DOLocationID=169, passenger_count=1.0, trip_distance=1.19, tip_amount=0.25) 1
Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0)Row(lpep_pickup_datetime='2019-10-01 00:26:02', lpep_dropoff_datetime='2019-10-01 00:39:58', PULocationID=112, DOLocationID=196, passenger_count=1.0, trip_distance=5.88, tip_amount=0.0) 1

7. Most popular destination

199 / 199 correct (100.0%)

Answer Count
74 108
DOLocationID = 74 6
East Harlem North 5
DOLocationID 74 2
Manhattan 2
JFK Airport 2
DOLocationID: 74 2
74 - Manhattan-East Harlem North 1
|{2024-03-15 12:00:00, 2024-03-15 12:05:00}|74 |35478| 1
The most popular destination was with 74 DOLocation ID as figure above. 1
DOLocationID : 74 1
74|Manhattan|East Harlem North| Boro Zone| 1
lpep_pickup_datetime 1
DOLocationID='74' 1
112 1
LocationID : 74 - Zone : East Harlem North 1
Topic 1
{2024-03-19 03:10:00, 2024-03-19 03:15:00}|74 |17733| 1
DOLocationID 74 is the most popular with 35482 total count 1
Destination with DOLocationID 74 is most popular. 1
74 or East Harlem North 1
DOLocationID=74 1
"Manhattan" which with DOLocationID = 74 1
DOlocation:74 Count:17741 1
The most popular DOLocationId is 74 1
This one I couldn't solve 1
74 = Manhattan 1
DOLocationID - 74, count - 106446 1
DOLocationID = 74, Count = 170137 1
166 1
DOLocationID 74 trip_coun 300599 1
window |DOLocationID|count| +------------------------------------------+------------+-----+ |{2024-03-17 20:25:00, 2024-03-17 20:30:00}|74 |35497| 1
East Harlem North (DOLocationID 74) 1
Stuck 1
East Harlem North (LocationID: 74) 1
89 1
DOLocationID 42 1
1 1
East Harlem North. 1
DOLoationID: 74 1
{2024-03-15 12:00:00, 2024-03-15 12:05:00}|74 |35478 1
ID - 74 1
74 window |DOLocationID|count| +------------------------------------------+------------+-----+ |{2024-03-28 15:25:00, 2024-03-28 15:30:00}|74 |50527| |{2024-03-28 15:25:00, 2024-03-28 15:30:00}|42 |45348| |{2024-03-28 15:25:00, 2024-03-28 15:30:00}|41 |39984| |{2024-03-28 15:25:00, 2024-03-28 15:30:00}|75 |36587| |{2024-03-28 15:25:00, 2024-03-28 15:30:00}|129 |33798| |{2024-03-28 15:25:00, 2024-03-28 15:30:00}|7 |32634| 1

Calculated: 13 October 2024, 17:46