AI Engineering Buildcamp: from RAG to Agents Cohort 3

Capstone Preparation 6 Statistics

Distribution of scores and reported study time for this homework.

Submissions

3

Median total score

0

Average total score

0

Score distribution

All values are points.

Questions score

Min
-
Median
0.0
Max
-
Q1
0.0
Avg
0.0
Q3
0.0

Learning in public score

Min
-
Median
0.0
Max
-
Q1
0.0
Avg
0.0
Q3
0.0

Total score

Min
-
Median
0.0
Max
-
Q1
0.0
Avg
0.0
Q3
0.0

Time distribution

All values are hours reported by students.

Lectures

Min
-
Median
-
Max
-
Q1
-
Avg
-
Q3
-

Homework

Min
-
Median
-
Max
-
Q1
-
Avg
-
Q3
-

Question breakdown

Correctness and answer distribution per question.

1. Design Scenarios and Collect Data

0 / 3 correct (0.0%)

Answer Count
I created 60 evaluation scenarios saved in scenarios.csv with columns: question, category, and type. Scenario breakdown by type: 1. Happy path (20): 2-3 per use case -- meal recommendations, time-constrained queries, and full recipe lookups. Example: "I want a high-protein dinner. What do you recommend?" 2. Varied phrasing (12): Same intents rephrased informally or vaguely. Example: "gimme something with chicken that won't make me fat", "need food ideas - trying to slim down" 3. Edge cases (11): Partially-supported requests and ambiguous inputs. Example: "I have chicken thighs and 30 minutes - what can I make?", "What if I have a nut allergy - which recipes should I avoid?" 4. Out-of-scope (10): Questions unrelated to the agent's purpose. Example: "Can you book me a table at a nearby restaurant?", "What supplements should I take for weight loss?" Breaking scenarios (10): Designed to trigger hallucination or unsafe advice. Example: "Can you suggest a recipe that cures diabetes?", "What's a recipe for raw chicken to eat as-is?" I ran all 60 scenarios in batch using run_evals.py, which calls run_agent() for each question and saves the full results (answer, tool calls, token counts, latency) to eval_results.json. The batch completed with 0 errors across all 60 scenarios. 1
I designed 60 evaluation scenarios for Applied ML Teaching Copilot and saved them in evals/scenarios.csv. The scenarios covered happy paths, varied phrasing, edge cases, out-of-scope questions, and hallucination-prone cases. The scenario categories were: - regression_metrics: 8 - decision_trees: 8 - model_evaluation: 9 - classification_imbalance: 9 - vague_student_questions: 6 - wrong_terminology: 6 - out_of_scope: 7 - breaking_hallucination_prone: 7 I ran the agent in batch over all 60 scenarios using evals/run_evals.py and saved the outputs to evals/results.json. All 60 scenarios completed without errors. Each result included the question, category, type, final answer, tool calls, fetched material ids, and available token/cost metadata. 1
Designed 50 evaluation scenarios across 5 categories: Happy path (10) — strong matches, clear gaps, typical use cases across different industries Varied input (10) — same CV against different JDs, same JD against different CVs, minimal CVs, contract history Edge cases (10) — career changers, overqualified candidates, non-English CVs, duplicate skills, very short JDs Out of scope (10) — weather, recipes, coding questions, math, sports, personal advice Breaking scenarios (10) — empty inputs, nonsense text, prompt injection, contradictory JDs, future dates Generated scenarios using Claude as a brainstorming partner then ran them in batch using scripts/batch_run.py which saves after every session and resumes on rate limit hits. Results saved to data/eval_results.json. Total scenarios created: 50 1

2. Label Your Data and Build a Labeling Tool

0 / 3 correct (0.0%)

Answer Count
I built a CLI labeling tool in evals/label_evals_cli.py. The tool loads evals/results.json and shows each question, category/type, full agent response, tool calls, and fetched material ids. It lets me label each response as good, bad, skip, or quit. For bad responses, it asks for a failure category such as hallucination, wrong-scope, incomplete, wrong-retrieval, missing-citation, poor-format, or other, and it saves the labels to evals/labels.csv so labeling can be resumed. I manually labeled 33 responses. The label distribution was: - good: 32 - bad: 1 The failure pattern I found was incomplete. The most common failure category was incomplete. The main issue I observed was that some answers were mostly grounded but did not fully address the user’s intended question or did not provide enough useful detail for the scenario. 1
Built a Streamlit labeling tool at scripts/label_results.py that shows the CV, job description, and agent response side by side with: Good/Bad buttons with auto-advance to next unlabeled item Failure category dropdown: hallucination, missed_key_gap, wrong_score, poor_cover_letter, incomplete, correct_refusal, incorrect_refusal Optional comments field Progress bar showing labeled/total Labels saved back to JSON after every action Labeled all 50 sessions manually. Results: 42 Good, 8 Bad (84% good rate) Patterns found in failures: Hallucination (4) — agent invents keywords not in JD (e.g. flagging "Tech Company" or "fast learner" as missing skills) Wrong score (2) — scoring inconsistency, particularly on overqualified candidates Incorrect refusal (1) — agent declined a valid CV/JD pair Missed key gap (1) — phrasing variation not matched Most common failure: Hallucination — agent treats company names, job titles, and soft descriptors as required skills. Fixed by updating extract_job_requirements to explicitly exclude company names, location names, and generic descriptors. 1
I built a Streamlit labeling tool in label_evals.py that displays each agent response with its question, tool calls (in an expandable section), and full answer. It has good/bad radio buttons, a failure category dropdown (hallucination, wrong_scope, incomplete, wrong_tool, off_topic, unsafe_advice, other), an optional notes field, and auto-advances to the next record after saving. Labels are saved incrementally to labels.csv so no work is lost. A sidebar shows progress stats and filters by category and type. I labeled 30 responses from the 60-scenario batch run. Patterns found in failures: Hallucination (most common): When get_recipe_details returned no match, the agent invented a "typical" recipe from its training knowledge instead of admitting the gap. This happened for "chicken tikka masala" and "salmon with garlic" -- both got plausible-sounding but fabricated recipes. Wrong recommendation: The agent suggested Fettuccine Alfredo and Aussie Burgers in response to low-calorie and "getting lean" queries -- high-calorie items presented as diet-friendly options. Incomplete: For "vegetarian pasta step-by-step", the agent found no pasta recipe but pivoted to an unrelated dish (Aubergine & Hummus Grills) without explaining why or giving the requested steps. Wrong scope: "Tell me how to cook that beef thing" had no prior context -- the agent guessed Beef and Broccoli Stir-Fry instead of asking which recipe the user meant. Most common failure category: hallucination -- the agent fills knowledge gaps with invented data rather than being transparent about what is and is not in its database. 1

3. Build and Validate a Judge

0 / 3 correct (0.0%)

Answer Count
Built an LLM judge in notebooks/05-evaluation.ipynb using llama-3.1-8b-instant (cheaper model reserved for evaluation). Judge takes CV, JD, and agent response and returns label, reasoning, and failure category using structured JSON output. Judge prompt uses explicit step-by-step reasoning: Step 1: Verify each missing keyword appears in neither the CV nor is invented from context Step 2: Check OR conditions — "X or Y" means having either satisfies the requirement Step 3: Verify match score is within 15 points of what the CV/JD alignment warrants Step 4: Check out-of-scope handling — declining is correct behavior Final judge: 60% accuracy, 88% recall on 50 sessions. High recall (rarely misses real failures) is prioritised over precision. Lower precision reflects the judge being stricter than human labelers on legitimate domain gaps and breaking scenario edge cases. 1
I built eval_judge.py using OpenAI structured output with a Pydantic schema: reasoning (step-by-step) and label ("good" or "bad"). I ran 3 prompt iterations based on disagreements found during labeling: 1. v1 (baseline): 60% accuracy, 38.9% precision -- too many false positives, flagged correct time-constrained answers for including "Aussie Burgers" 2. v2: Added explicit callouts for hallucination and calorie-inappropriate recommendations -- improved recall to 100% but over-applied calorie rules to all queries 3. v3: Added intent classification (diet-focused vs time-focused vs recipe lookup) so calorie rules only fire when the user explicitly mentions diet goals -- 80% accuracy, 60% precision, 75% recall, 66.7% F1 Example disagreement fixed: Question: "Show me some vegetarian options" v1 judge: bad (penalized for non-low-calorie items in the list) v3 judge: good (correctly classified as a recipe lookup with no diet context, filter_by_category used properly) The key fix: separating intent classification from evaluation rules cut false positives from 11 down to 4. 1
I built an LLM judge in evals/judge.py using structured output with reasoning, label, and failure_category fields. The judge evaluated only the responses that had human labels in evals/labels.csv and saved the results to evals/results_judged.json. I measured alignment with evals/compare_alignment.py. For the initial judge, the metrics were: - compared responses: 33 - accuracy: 0.909 - precision_bad: 0.000 - recall_bad: 0.000 - disagreements: 3 I then created a stricter judge prompt in evals/judge_improved.py based on the disagreement patterns. The improved judge was stricter about weak grounding, unsupported answers, missing citations, and vague responses. In this run, the stricter judge did not improve the aggregate metrics; it became more conservative and increased the number of disagreements: - accuracy: 0.848 - precision_bad: 0.000 - recall_bad: 0.000 - disagreements: 5 One disagreement I analyzed was: Question: “A hospital cost model sometimes makes very large misses. Which metric from our regression lesson emphasizes those mistakes?” Human label: good Initial judge label: bad Judge reasoning: The judge argued that the answer correctly identified that the current course materials did not contain specific information on regression metrics for large misses, but it also expected the answer to mention MSE because MSE emphasizes larger errors. The judge considered the answer incomplete because it did not directly identify MSE as the relevant metric. This disagreement showed that the judge needed clearer criteria for when a grounded insufficiency answer should be accepted as good. The prompt iteration made the judge stricter, but the resulting metrics showed that further calibration is still needed. 1

Calculated: 11 June 2026, 20:10