AI Engineering Buildcamp: from RAG to Agents Cohort 3

Capstone Preparation 5 Statistics

Distribution of scores and reported study time for this homework.

Submissions

5

Median total score

0

Average total score

0

Score distribution

All values are points.

Questions score

Min
-
Median
0.0
Max
-
Q1
0.0
Avg
0.0
Q3
0.0

Learning in public score

Min
-
Median
0.0
Max
-
Q1
0.0
Avg
0.0
Q3
0.0

Total score

Min
-
Median
0.0
Max
-
Q1
0.0
Avg
0.0
Q3
0.0

Time distribution

All values are hours reported by students.

Lectures

Min
-
Median
-
Max
-
Q1
-
Avg
-
Q3
-

Homework

Min
-
Median
-
Max
-
Q1
-
Avg
-
Q3
-

Question breakdown

Correctness and answer distribution per question.

1. Monitoring Platform Setup

0 / 5 correct (0.0%)

Answer Count
Logfire — familiar from course content, OpenTelemetry-based. Dashboard shows: agent run spans with full tool call traces, token usage per LLM call, match scores from score_cv, and session duration. 1
I chose Pydantic Logfire because of it's simplicity and easy setup. I would like to follow the whole process the agents goes through when answering a user question. 1
I chose Logfire for monitoring because I had not used it before and wanted to gain experience with it. It is quick to implement, provides a lot of useful tracking information out of the box, and is well suited for rapid prototyping and debugging during the early development phase. In the dashboard, I would mainly like to see tool calls, token usage, latency, and user feedback. For a more production-ready setup, I would consider switching later to a combination of PostgreSQL and Grafana, since I already have experience with these tools and they offer more flexibility and scalability for long-term monitoring and analytics. 1
I built a custom file-based monitoring system using Python instead of Logfire or Langfuse. Each agent run saves a JSON trace to data/traces/ containing everything needed to analyze and replay the interaction. What is tracked in each trace: Traces: trace_id, session_id, timestamp, question, answer Tool calls: ordered list of every tool the agent called with its exact arguments Token usage: input_tokens and output_tokens accumulated across all LLM iterations in the agent loop Performance: duration_seconds per turn User feedback: thumbs up (1) or thumbs down (-1) The Streamlit sidebar acts as the live dashboard, showing turns this session, tokens used, average response time, and all-time feedback counts. I chose a custom solution over Logfire because it requires no external account, the trace files are plain JSON so they are easy to query and feed into evaluation next week, and the setup is fully contained in the repo. 1
I used Logfire as the monitoring platform for Applied ML Teaching Copilot. I chose Logfire because it provides observability for AI applications through traces, spans, LLM calls, and tool execution logs. It also integrates well with Python projects and allows me to inspect what the agent did during each interaction. In the dashboard, I wanted to see complete traces for each agent run, including the user query, OpenAI calls, tool calls, fetched course-material ids, final answer previews, token usage, and feedback events. I also wanted to group multiple interactions under a session-level span so that a teaching session could be inspected as one complete flow instead of isolated calls. In my run, Logfire captured spans such as teaching_copilot_session, teaching_copilot_run, OpenAI Responses API calls, and user_feedback events. 1

2. Interaction Data Collection Plan

0 / 5 correct (0.0%)

Answer Count
I planned to collect the following interaction data: - user queries - tool calls made by the agent - number of tool calls per interaction - tool call patterns, such as search_course_materials followed by get_course_material - fetched course-material ids when available - final answer previews - whether the answer was a grounded insufficiency response - token usage per interaction - simulated user feedback events I plan to use this data for evaluation next week. The traces will help identify whether the agent is using tools correctly, whether it retrieves the right course materials, and whether it avoids hallucinating when the knowledge base is insufficient. Feedback events can help separate useful answers from problematic interactions, while token usage helps estimate the operational cost of running and evaluating the agent. 1
Match score per session — tracking scoring consistency over time Token usage per LLM call — estimating cost at scale Tool call sequence — detecting when agent skips or repeats tools Session duration — identifying slow sessions hitting rate limits 1
I would like to track: - Feedback from the user: was the question answered correctly? - Costs and time it took to answer the question - The final SQL code the agent returned 1
I would like to collect data about tool performance, token usage, and user interaction. In particular, I want to measure how long individual tool calls take in order to identify areas where latency is still too high. I also want to track whether the tools are being called in the correct order and how high the token costs are for different requests. In addition, I want to understand where the agent or specific tools are not performing as expected. I also want to analyze whether the prompts are being executed correctly and identify situations where further prompt engineering or optimization may be necessary. 1
Data collected per turn and how it will be used: Question text - cluster common query types, find gaps in the recipe database (e.g. if many users ask for calorie counts but the data has none) Tool call sequence - verify the agent picks the right tool in the right order; a time-constrained query should always trigger filter_by_max_cook_time, not just search_recipes Tool arguments - check that extracted values are correct, e.g. the max_minutes argument actually matches what the user said Token counts - track cost per query type; identify unexpectedly expensive patterns (combined-constraint queries used 15k input tokens vs 600 for out-of-scope ones) Duration - catch latency regressions across model or prompt changes Thumbs feedback - binary label for whether the answer satisfied the user; this becomes the ground-truth signal for evaluation next week 1

3. Running and Collecting Data

0 / 5 correct (0.0%)

Answer Count
Sessions collected: 6 sessions collected covering strong match, weak match, career changer, overqualified candidate, logistics domain match, and a quick test. Average session uses 7,955 tokens (~$0 on Groq free tier, ~$0.04 on GPT-4o). Patterns noticed: senior CV against junior role scored 70 (too generous), weak match correctly scored 30, sessions with the <function= recovery handler triggered use 5 LLM calls vs normal 2-3. 1
sessions collected: 10 average costs: 0,01 $ average tokens: 80.000 issue: The first tool call takes almost 2 minutes. This is because the system first downloads a transcript of a YouTube video, and the agent generates a summary of the core concepts so that it can ask learning-specific questions based on that summary. Since these transcripts can sometimes be very long, the process takes a correspondingly long time. I’m considering whether I should just chunk and index the transcript and let the user decide what to talk about without any suggestions. But I’m not sure yet how to handle. 1
I ran 8 sessions covering all major query types: high-protein search, time-constrained, category filter, full recipe request, combined constraints, and one out-of-scope request. Sessions collected: 8 Typical token usage: ~5,750 input / ~330 output per turn, roughly $0.001 per turn at gpt-4o-mini pricing Patterns noticed: The combined vegetarian + time query triggered filter_by_category twice before calling filter_by_max_cook_time, using 15k input tokens instead of the expected 3k. The agent over-calls category filters when constraints are combined, which is a clear fix target. The out-of-scope booking request correctly made zero tool calls and used only 607 input tokens, showing the agent does not waste calls on requests it cannot handle. Full recipe requests using get_recipe_details were the fastest turns at around 8-10 seconds since they only need one tool call. 1
I ran one monitored session with three interactions using Applied ML Teaching Copilot. The collected interactions were: 1. "When should I use MAE instead of MSE in a regression problem?" The agent made 2 tool calls and produced a grounded answer comparing MAE and MSE. 2. "Give me a short study guide about decision trees." The agent made 4 tool calls and produced a structured study guide based on the available course materials. 3. "Can you explain convolutional neural networks using the course materials?" The agent made 1 tool call and correctly returned a grounded insufficiency response because the current course materials do not contain enough information about CNNs. The monitored session collected 3 total interactions and used approximately 4,524 total tokens. The script did not compute an exact cost because it did not separate input and output tokens for the monitored session, but with gpt-4o-mini the approximate cost was well below one cent. The main pattern I noticed was that normal Applied ML questions triggered the expected search and fetch behavior, while the out-of-scope CNN query triggered only search_course_materials and then returned an insufficiency response instead of hallucinating. This is useful for next week’s evaluation because it gives both successful interactions and an edge case to evaluate. 1

Calculated: 11 June 2026, 20:10