Capstone Preparation 4

1. Think About Test Scenarios (2 points)

Scenario 1 - Study plan after progress exists "I have 5 hours this weekend. What should I focus on?" Expected tool calls, in order: 1. get_progress() - check what's done 2. search_knowledge_base("AI engineering", exclude_completed=True, num_results=5) Expected output: 3–4 resources not in completed, with realistic time allocation adding up to ~5 hours. Likely failure: agent skips get_progress and goes straight to search, then recommends resources the user already finished. We've seen it call both correctly in smoke tests, but it's non-deterministic, worth testing repeatedly. Scenario 2 - "I finished X, what's next?" "I just finished the Karpathy video series. What should I study next?" Expected tool calls: 1. update_progress("Neural Networks: Zero to Hero", "completed") - mark it done 2. get_progress() 3. search_knowledge_base("deep learning next steps after neural networks", exclude_completed=True) Likely failure - two of them: * Agent skips update_progress entirely and just searches, never recording the completion * Resource name mismatch: user says "Karpathy video series", KB title is "Neural Networks: Zero to Hero". Agent might pass the wrong string to update_progress, leaving the resource un-marked in progress.json. This is the scenario I'm most worried about. It's the core progress-tracking loop and it has a fuzzy matching problem baked in. Scenario 3 - Topic not in KB, agent should say so honestly "What do I have on transformer architecture in my knowledge base?" Expected tool calls: 1. search_knowledge_base("transformer architecture") Expected output: return whatever is relevant (there are some LLM resources), or clearly say "nothing specific on transformers" if results are weak. Likely failure: the agent returns loosely related resources (prompt engineering, LLMs) and presents them as if they cover transformer architecture - confidently recommending something that doesn't match the query. This is the hallucination-adjacent failure you flagged as worst-case.

2. Implement a Test (2 points)

3. Add an LLM Judge Test (2 points)

I added an LLM judge test in: tests/test_judge.py The judge test was: test_judge_mae_vs_mse_grounding It ran the agent on the question: "When should I use MAE instead of MSE in a regression problem?" Then it evaluated the agent output with specific natural-language criteria: 1. The agent called search_course_materials before get_course_material. 2. The answer compared MAE and MSE using the course material rather than unsupported general knowledge. 3. The answer stated that MAE is less sensitive to outliers or easier to interpret in target units. 4. The answer stated that MSE penalizes large errors more strongly. 5. The answer cited material id aml-001. The LLM judge test passed as part of the full pytest run. The run also tracked OpenAI token usage: gpt-4o-mini: 5685 input tokens, 1072 output tokens