Homework 1: Pre-Requisites (Docker, Terraform, SQL)

Score: 9 = 7 (questions) + 1 (FAQ) + 1 (learning in public)

Homework URL: View submission

Learning in public links: Show

• https://www.pintonista.com/running-postgres-and-pgadmin-with-docker-compose/

Homework 2: Mage

Score: 7 = 6 (questions) + 0 (FAQ) + 1 (learning in public)

Homework URL: View submission

Learning in public links: Show

• https://www.jonahboliver.com/blog/de-zc-w2#configuring-postgresql-connection

Workshop 1: Data Ingestion

Score: 6 = 4 (questions) + 0 (FAQ) + 2 (learning in public)

Homework URL: View submission

Learning in public links: Show

• https://dlthub.com/docs/tutorial/load-data-from-an-api
• https://dlthub.com/docs/dlt-ecosystem/destinations/bigquery

Homework 3: Data Warehouse

Score: 9 = 6 (questions) + 0 (FAQ) + 3 (learning in public)

Homework URL: View submission

Learning in public links: Show

• https://www.jonahboliver.com/blog/de-zc-w3#exporting-taxi-data-into-google-cloud-storage
• https://medium.com/data-nuggets/create-and-load-partitioned-and-clustered-table-in-bigquery-using-python-1a2eab313be7
• https://medium.com/google-colab/a-better-way-to-use-google-cloud-from-colab-bb93f88b5021

Homework 4: Analytics Engineering

Score: 2 = 2 (questions) + 0 (FAQ) + 0 (learning in public)

Homework URL: View submission

Homework 5: Batch processing

Score: 8 = 6 (questions) + 0 (FAQ) + 2 (learning in public)

Homework URL: View submission

Learning in public links: Show

• https://sparkbyexamples.com/pyspark/
• https://www.geeksforgeeks.org/find-minimum-maximum-and-average-value-of-pyspark-dataframe-column/

Workshop 2: Streaming with SQL

Score: 3 = 3 (questions) + 0 (FAQ) + 0 (learning in public)

Homework URL: View submission

Homework 6: Streaming

Score: 10 = 6 (questions) + 0 (FAQ) + 4 (learning in public)

Homework URL: View submission

Learning in public links: Show

• https://sparkbyexamples.com/pyspark/
• https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
• https://medium.com/@suffyan.asad1/spark-leveraging-window-functions-for-time-series-analysis-in-pyspark-03aa735f1bdf
• https://spark.apache.org/docs/2.2.1/sql-programming-guide.html

Public Leaderboard

Score: 0 = 0 (questions) + 0 (FAQ) + 0 (learning in public)

Project Attempt 2

Project score: 14 Passed

Score: 29 = 14 (project) + 9 (peer review) + 5 (learning in public / project) + 0 (learning in public / peer review) + 1 (FAQ)

Project URL: View project

Learning in public links: Show

• https://jedri.medium.com/run-docker-compose-on-gce-a5a49f64225e https://medium.com/cloudnloud/getting-started-with-google-cloud-compute-engine-and-docker-3370857b04b0
• https://github.com/norberte/DS_professional_development/blob/master/DS%20interviews/NYC%20Yellow%20Taxi.ipynb
• https://medium.com/@kangzhiyong1999/using-mage-as-the-workflow-orchestration-tools-ef7547fb18fc
• https://medium.com/@ofelipefernandez/mage-etl-to-google-cloud-and-bigquery-72d79e9b5f33
• https://www.analyticsvidhya.com/blog/2021/01/exploratory-data-analysis-on-nyc-taxi-trip-duration-dataset/