LLM Zoomcamp 2026

Homework 4: Evaluation Statistics

Distribution of scores and reported study time for this homework.

Submissions

374

Median total score

6

Average total score

6

Score distribution

All values are points.

Questions score

Min
1
Median
6.0
Max
6
Q1
5.0
Avg
5.2
Q3
6.0

Learning in public score

Min
-
Median
0.0
Max
7
Q1
0.0
Avg
0.7
Q3
1.0

Total score

Min
1
Median
6.0
Max
14
Q1
5.0
Avg
5.9
Q3
6.0

Time distribution

All values are hours reported by students.

Lectures

Min
0.0
Median
4.0
Max
50.0
Q1
3.0
Avg
5.7
Q3
6.0

Homework

Min
1.0
Median
3.0
Max
24.0
Q1
2.0
Avg
3.5
Q3
4.0

Question breakdown

Correctness and answer distribution per question.

1. When generating questions for the first 3 lesson pages, what is the average number of input tokens across these 3 calls?

354 / 374 correct (94.7%)

1 140 8 (2.1%)
2 1400 354 (94.7%)
3 14000 10 (2.7%)
4 140000 1 (0.3%)

2. After running text_search for the first ground truth question, what is the filename of the first result?

321 / 374 correct (85.8%)

1 01-agentic-rag/lessons/01-intro.md 47 (12.6%)
2 01-agentic-rag/lessons/03-rag.md 321 (85.8%)
3 01-agentic-rag/lessons/13-function-calling.md 5 (1.3%)
4 01-agentic-rag/lessons/10-rag-next-steps.md 0 (0.0%)

3. After running vector_search for the same question, what is the filename of the first result?

331 / 374 correct (88.5%)

1 01-agentic-rag/lessons/01-intro.md 331 (88.5%)
2 01-agentic-rag/lessons/03-rag.md 24 (6.4%)
3 04-evaluation/lessons/11-evaluation-intro.md 12 (3.2%)
4 04-evaluation/lessons/12-rag-answers.md 4 (1.1%)

4. After evaluating text_search on the ground truth, what is the Hit Rate?

336 / 374 correct (89.8%)

1 0.55 5 (1.3%)
2 0.66 11 (2.9%)
3 0.76 336 (89.8%)
4 0.88 18 (4.8%)

5. After evaluating vector_search on the ground truth, what is the MRR?

304 / 374 correct (81.3%)

1 0.35 2 (0.5%)
2 0.45 17 (4.5%)
3 0.55 304 (81.3%)
4 0.65 46 (12.3%)

6. When evaluating hybrid_search for k values 1, 50, 100, and 200, which k gives the best MRR?

300 / 374 correct (80.2%)

1 1 300 (80.2%)
2 50 58 (15.5%)
3 100 9 (2.4%)
4 200 1 (0.3%)

Calculated: 22 July 2026, 19:37