AI Engineering Buildcamp: from RAG to Agents Cohort 3

Capstone Preparation 4 Statistics

Distribution of scores and reported study time for this homework.

Submissions

7

Median total score

0

Average total score

0

Score distribution

All values are points.

Questions score

Min
-
Median
0.0
Max
-
Q1
0.0
Avg
0.0
Q3
0.0

Learning in public score

Min
-
Median
0.0
Max
-
Q1
0.0
Avg
0.0
Q3
0.0

Total score

Min
-
Median
0.0
Max
-
Q1
0.0
Avg
0.0
Q3
0.0

Time distribution

All values are hours reported by students.

Lectures

Min
-
Median
-
Max
-
Q1
-
Avg
-
Q3
-

Homework

Min
1.0
Median
3.0
Max
6.0
Q1
1.0
Avg
3.2
Q3
5.0

Question breakdown

Correctness and answer distribution per question.

1. Think About Test Scenarios

0 / 7 correct (0.0%)

Answer Count
Tool call order — When a user provides a CV and job description, the agent must call all four tools in the correct sequence: extract_job_requirements → score_cv → suggest_improvements → generate_cover_letter. If any tool is skipped or called out of order the output is meaningless. Gap report quality — The final response must include a numeric match score, at least one specific missing keyword, actionable suggestions mentioning specific skills from the JD (like Tableau or numpy), and a tailored cover letter. During testing we discovered the agent was calling generate_cover_letter but dropping the cover letter from the final output — the judge caught this and we fixed the agent instructions. Out of scope input — When a user asks something unrelated to CV analysis (e.g. "What's the weather today?"), the agent should respond without calling any ATS tools. Missing keywords in suggestions — Keywords identified as missing by score_cv should appear explicitly in the suggestions from suggest_improvements. Suggestions should not be generic advice that ignores what was actually missing. 1
1. Does the agent download and process the YouTube transcript? 2. Is the order and number of tool calls in the first iteration correct? Are too many tool calls being made? 3. Does the agent provide the core concepts of the youtube video and suggest questions based on that summary? 1
Five scenarios covering different tool paths and failure modes: High-protein query - user asks for protein-rich meals. The agent should call search_recipes first, and every recipe in the answer should include cooking time in minutes. Time-constrained query - user says "I only have 15 minutes." The agent must call filter_by_max_cook_time (not just a search) and the max_minutes argument must be 15 or less. I noticed that without this check the agent sometimes just called search_recipes with the word "quick" and recommended dishes that take 30+ minutes. Full recipe request - user asks for step-by-step cooking instructions. The agent should call get_recipe_details to retrieve the full recipe, and the answer should contain actual ingredient quantities, not a generic summary. Out-of-scope request - user asks to book a restaurant table. The agent cannot do this and must not claim otherwise. The answer must not contain "booked" or "reservation confirmed". Category filter query - user says "only vegetarian dishes." The agent should call filter_by_category with category="Vegetarian" so off-category recipes are excluded from the start. 1
I defined the following test scenarios for Applied ML Teaching Copilot: 1. MAE vs MSE grounded answer User question: "When should I use MAE instead of MSE in a regression problem?" Expected behavior: the agent should call search_course_materials first, then get_course_material for the relevant course record. The answer should compare MAE and MSE, mention that MAE is easier to interpret or less sensitive to outliers, mention that MSE penalizes large errors more strongly, and cite aml-001. 2. Decision tree study guide User question: "Give me a short study guide about decision trees." Expected behavior: the agent should search the course materials, fetch relevant records if needed, and return a structured study guide grounded in the available material. The answer should mention decision trees and cite at least one material id. 3. Out-of-scope CNN query User question: "Can you explain convolutional neural networks using the course materials?" Expected behavior: the agent should search the course materials. If no relevant material is found, it should clearly say that the current course-material knowledge base is insufficient. It should not provide a general explanation from model knowledge. 4. Tool behavior and order For course-topic questions, the agent should call search_course_materials before get_course_material. This verifies that the agent searches compact metadata first and only fetches full records after identifying relevant material. 1
Scenario 1 - Study plan after progress exists "I have 5 hours this weekend. What should I focus on?" Expected tool calls, in order: 1. get_progress() - check what's done 2. search_knowledge_base("AI engineering", exclude_completed=True, num_results=5) Expected output: 3–4 resources not in completed, with realistic time allocation adding up to ~5 hours. Likely failure: agent skips get_progress and goes straight to search, then recommends resources the user already finished. We've seen it call both correctly in smoke tests, but it's non-deterministic, worth testing repeatedly. Scenario 2 - "I finished X, what's next?" "I just finished the Karpathy video series. What should I study next?" Expected tool calls: 1. update_progress("Neural Networks: Zero to Hero", "completed") - mark it done 2. get_progress() 3. search_knowledge_base("deep learning next steps after neural networks", exclude_completed=True) Likely failure - two of them: * Agent skips update_progress entirely and just searches, never recording the completion * Resource name mismatch: user says "Karpathy video series", KB title is "Neural Networks: Zero to Hero". Agent might pass the wrong string to update_progress, leaving the resource un-marked in progress.json. This is the scenario I'm most worried about. It's the core progress-tracking loop and it has a fuzzy matching problem baked in. Scenario 3 - Topic not in KB, agent should say so honestly "What do I have on transformer architecture in my knowledge base?" Expected tool calls: 1. search_knowledge_base("transformer architecture") Expected output: return whatever is relevant (there are some LLM resources), or clearly say "nothing specific on transformers" if results are weak. Likely failure: the agent returns loosely related resources (prompt engineering, LLMs) and presents them as if they cover transformer architecture - confidently recommending something that doesn't match the query. This is the hallucination-adjacent failure you flagged as worst-case. 1
Searches contract before computing any deadline Does not assume federal holidays for non-federal contracts Flags "days" ambiguity after checking contract definitions, not before Declines off-topic questions and stays in scope Calls list_contracts or asks which contract before computing Does not re-ask for information the rep already provided Uses tools to find contract language, never asks rep to paste it Understands numeric contract selection ("3" = third contract from list) Asks rep which days their bargaining unit works before computing working-day deadlines Computes correctly when rep explicitly states a 7-day work schedule 1
- Always have a very high-confidence of the output before giving it back to the user - Make sure follow-up questions are related to database querying - Never make up random numbers 1

2. Implement a Test

0 / 7 correct (0.0%)

Answer Count
https://github.com/Amar-Ag/ats-gap-analyser/blob/main/tests/test_agent.py 1
https://github.com/katjaweb/gapfinder/blob/main/tests/test_agent.py 1
See tests/test_agent.py - 9 deterministic pytest tests, no LLM judge. 1
I implemented pytest tests in: tests/test_agent.py 1
https://github.com/wesleytanjiale/ai-learning-os/blob/main/notebooks/tests/test_agent.py#L18 1
Done 1
https://github.com/larsvasseldonk/relational-rag/blob/main/src/agent/tests/test_agent.py 1

3. Add an LLM Judge Test

0 / 7 correct (0.0%)

Answer Count
await assert_criteria(result, [ "makes at least 2 tool calls", "provides key concepts of the video in the output", "asks the user if they want to explore the concepts", "suggests questions for the concepts" ]) 1
- the response includes a numeric match score between 0 and 100 - the response lists at least one specific missing keyword from the job description - the response includes a cover letter addressed to the specific role not a generic template - the suggestions mention specific skills from the job description such as Tableau or numpy not generic advice 1
I added an LLM judge test in: tests/test_judge.py The judge test was: test_judge_mae_vs_mse_grounding It ran the agent on the question: "When should I use MAE instead of MSE in a regression problem?" Then it evaluated the agent output with specific natural-language criteria: 1. The agent called search_course_materials before get_course_material. 2. The answer compared MAE and MSE using the course material rather than unsupported general knowledge. 3. The answer stated that MAE is less sensitive to outliers or easier to interpret in target units. 4. The answer stated that MSE penalizes large errors more strongly. 5. The answer cited material id aml-001. The LLM judge test passed as part of the full pytest run. The run also tracked OpenAI token usage: gpt-4o-mini: 5685 input tokens, 1072 output tokens 1
def test_study_plan_output_quality(agent, progress_with_karpathy_done): """ LLM judge evaluates whether the study plan answer meets quality criteria that are hard to assert with simple string checks. Criteria are specific and concrete - each one checks exact behaviour, not vague properties like 'is helpful'. """ user_prompt = "I have 5 hours this weekend. What should I focus on?" result = agent(user_prompt) print(f"\nanswer: {result.answer}") print(f"\ntool_calls: {result.tool_calls}") assert_criteria(user_prompt, result, [ "every resource title mentioned in the study plan exists verbatim in the " "knowledge base — the agent must not invent or paraphrase resource names", "the time allocations listed in the study plan sum to approximately 5 hours " "(between 4 and 6 hours total) — the agent must respect the user's time constraint", "the resource 'Neural Networks: Zero to Hero' does not appear anywhere in the " "response, since the user has already completed it", ]) 1
See tests/test_judge.py. The criteria for the time-constrained test: assert_criteria(result, [ "the agent called filter_by_max_cook_time with a max_minutes value of 15 or less", "every recipe the agent recommends has a cooking time of 15 minutes or less", ]) The first criterion checks tool behavior (the right tool was called with the right argument). The second checks output correctness (the actual recommendations obey the constraint, not just the tool call). Both passed across all runs. 1
@pytest.mark.asyncio async def test_uses_tools_to_find_contract_language(agent): result = await run_agent_test( agent, "I filed a grievance on May 1st 2026 under the HUD AFGE contract. " "What is my Step 2 deadline?", ) tool_calls = collect_tools(result.new_messages()) await assert_criteria( result, [ "calls search_sections to find the grievance procedure article rather than asking the rep for it", "never asks the rep to paste or provide contract text", ], ) 1
"the SQL query only uses incidents reported in May 2025 by filtering on the dimdatum table", "the SQL query filters for 'Overlast' incidents using the 'hoofdsoort' column", "the SQL query counts the number of incidents by weekday", "the agent returns the weekday with the highest number of incidents.", "the agent does not filter on station or location type since the user did not ask for that", 1

Calculated: 11 June 2026, 20:10