Continuous Quality Monitoring for .NET AI Agents with Amazon Bedrock
Time to be announcedSpeakers
About this session
Most teams test an AI agent by chatting with it a few times, deciding it looks fine, and shipping it. That works until someone changes the prompt, swaps the model, or edits a tool definition, and the agent starts picking the wrong tool or giving confidently wrong answers. Nothing crashes. No error gets logged. The only signal is a support ticket a few days later.
This talk covers how to catch that kind of regression before it reaches users, using Amazon Bedrock AgentCore Evaluations. We'll build and instrument a .NET agent with OpenTelemetry, then run three demos live, scoring one session against built-in evaluators for correctness, helpfulness, and tool-use accuracy; adding an evaluation step to a GitHub Actions pipeline so a pull request fails automatically when agent quality drops; and checking evaluation scores against production traffic to catch drift after release. Attendees will walk away with working code and a pattern they can apply to their own .NET agents, regardless of which agent framework they're using, since AgentCore Evaluations works off OpenTelemetry traces rather than a specific SDK.