Homework 1: Introduction
- Submissions
- 580
- Completion
- 41.1%
- Median score
- 6
- Lecture time
- 3h
- Homework time
- 2h
- Total time
- 6h
LLM Zoomcamp 2024
LLM Zoomcamp 2024 Dashboard
Course-level participation, homework, and project statistics.
Course overview
- Total enrollments
- 1412
- Project submissions
- 221
- Project completion rate
- 13.4%
- Average total score
- 12.8
- Overall completion rate
- 26.0%
- Graduates
- 170
Project outcomes
- Score to pass
- 0
- Passed
- 177
- Failed
- 44
- Median project score
- 26
Homework statistics
Median time and score by homework. Hover values on desktop for interquartile ranges.
| Homework | Submissions | Completion | Lecture time | Homework time | Total time | Median score |
|---|---|---|---|---|---|---|
| Homework 1: Introduction | 580 | 41.1% | 3h | 2h | 6h | 6 |
| Saturn Cloud | 902 | 63.9% | - | - | - | - |
| Homework 2: Open-Source LLMs | 419 | 29.7% | 3h | 2h | 6h | 6 |
| Homework 3: Vector Search | 354 | 25.1% | 4h | 3h | 8h | 6 |
| Homework 4: Evaluation and monitoring | 288 | 20.4% | 4h | 2h | 7h | 6 |
| Homework 5: Ingestion pipeline | 174 | 12.3% | 2h | 3h | 6h | 6 |
| Workshop: Open-Source Data Ingestion for RAGs with dlt | 191 | 13.5% | 2h | 2h | 4h | 3 |
| LLM Zoomcamp 2024 Competition | 26 | 1.8% | - | - | - | 18 |
Saturn Cloud
- Submissions
- 902
- Completion
- 63.9%
- Median score
- -
- Lecture time
- -
- Homework time
- -
- Total time
- -
Homework 2: Open-Source LLMs
- Submissions
- 419
- Completion
- 29.7%
- Median score
- 6
- Lecture time
- 3h
- Homework time
- 2h
- Total time
- 6h
Homework 3: Vector Search
- Submissions
- 354
- Completion
- 25.1%
- Median score
- 6
- Lecture time
- 4h
- Homework time
- 3h
- Total time
- 8h
Homework 4: Evaluation and monitoring
- Submissions
- 288
- Completion
- 20.4%
- Median score
- 6
- Lecture time
- 4h
- Homework time
- 2h
- Total time
- 7h
Homework 5: Ingestion pipeline
- Submissions
- 174
- Completion
- 12.3%
- Median score
- 6
- Lecture time
- 2h
- Homework time
- 3h
- Total time
- 6h
Workshop: Open-Source Data Ingestion for RAGs with dlt
- Submissions
- 191
- Completion
- 13.5%
- Median score
- 3
- Lecture time
- 2h
- Homework time
- 2h
- Total time
- 4h
LLM Zoomcamp 2024 Competition
- Submissions
- 26
- Completion
- 1.8%
- Median score
- 18
- Lecture time
- -
- Homework time
- -
- Total time
- -
Assignment difficulty
Ranked by lowest median question score relative to the maximum achievable. Normalizing by question count makes homeworks of different lengths comparable; a lower percentage indicates a harder assignment.
| Rank | Homework | Median score | Score % | Completion |
|---|---|---|---|---|
| 1 | Homework 1: Introduction | 6 / 6 | 100.0% | 41.1% |
| 2 | Homework 2: Open-Source LLMs | 6 / 6 | 100.0% | 29.7% |
| 3 | Homework 3: Vector Search | 6 / 6 | 100.0% | 25.1% |
| 4 | Homework 4: Evaluation and monitoring | 6 / 6 | 100.0% | 20.4% |
| 5 | Workshop: Open-Source Data Ingestion for RAGs with dlt | 3 / 3 | 100.0% | 13.5% |
| 6 | Homework 5: Ingestion pipeline | 6 / 6 | 100.0% | 12.3% |
Question difficulty
Share of submitted answers that were correct, per question. A lower percentage points to a harder or more confusing question. Participation-only questions are excluded.
| Homework | Question | Correct | Answers | % correct |
|---|---|---|---|---|
| Homework 1: Introduction | Indexing the data | 551 | 580 | 95.0% |
| Searching: score for top record | 434 | 580 | 74.8% | |
| Filtering: 3rd document | 522 | 580 | 90.0% | |
| Prompt length | 462 | 580 | 79.7% | |
| Number of tokens | 452 | 580 | 77.9% | |
| Saturn Cloud | If you want access to SaturnCloud, write your email here | 0 | 902 | 0.0% |
| Homework 2: Open-Source LLMs | Downloading the weights. Size of the folder | 393 | 419 | 93.8% |
| Homework 3: Vector Search | Getting the embeddings model: First value | 342 | 354 | 96.6% |
| Search: Highest score | 341 | 354 | 96.3% | |
| Hit-rate for our search engine | 310 | 354 | 87.6% | |
| Hit-rate for Elasticsearch | 252 | 354 | 71.2% | |
| Homework 4: Evaluation and monitoring | Embedding first value | 278 | 288 | 96.5% |
| Dot product: 75 percentile | 276 | 288 | 95.8% | |
| Cosine 75 percentile | 271 | 288 | 94.1% | |
| Rouge 1 F | 272 | 288 | 94.4% | |
| Average Rouge | 269 | 288 | 93.4% | |
| Average Rouge 2 | 257 | 288 | 89.2% | |
| Homework 5: Ingestion pipeline | Number of documents | 170 | 174 | 97.7% |
| Chunking: number of questions | 167 | 174 | 96.0% | |
| Workshop: Open-Source Data Ingestion for RAGs with dlt | Rows in LanceDB | 130 | 191 | 68.1% |
| Running the Pipeline: Last edited time | 133 | 191 | 69.6% | |
| Ask the Assistant: PTO days | 141 | 191 | 73.8% | |
| LLM Zoomcamp 2024 Competition | Your kaggle leadearboard name | 0 | 26 | 0.0% |
| Your score (private) | 0 | 26 | 0.0% |
Submission timing
When homework submissions arrive relative to the deadline, across all assignments.
- A week or more early
- 619 (21.1%)
- 3-7 days early
- 754 (25.7%)
- 1-3 days early
- 516 (17.6%)
- Final day
- 630 (21.5%)
- After the deadline
- 415 (14.1%)
Engagement over time
Homework submissions per week, across all assignments. Bars are scaled to the busiest week.
- Jun 10, 2024
- 20
- Jun 17, 2024
- 815
- Jun 24, 2024
- 561
- Jul 01, 2024
- 287
- Jul 08, 2024
- 325
- Jul 15, 2024
- 316
- Jul 22, 2024
- 158
- Jul 29, 2024
- 107
- Aug 05, 2024
- 125
- Aug 12, 2024
- 64
- Aug 19, 2024
- 117
- Aug 26, 2024
- 13
- Sep 30, 2024
- 1
- Oct 14, 2024
- 25
Graduates
170 students completed the course and received certificates.