Prompt engineering has traditionally been manual trial-and-error. When AI agents degrade in production, teams make gut-feel changes and hope for the best. How do you know if a prompt change actually improves things, or just shifts failures elsewhere? We built an automated system that transforms production failures into quantifiable fixes. It starts by fetching examples of suboptimal sessions from our observability platform. The system analyzes these failures and clusters them into actionable patterns. For each pattern, it proposes specific prompt changes with before-and-after text. Then comes the critical step: validation. Only improvements that maintain or boost performance against our golden dataset make it into an automated pull request. Finally, the golden dataset itself is updated with representative samples of assessed failure. The result? We analyze all failures weekly and ship validated fixes—all without manual debugging. This approach transforms vague ""my AI isn't working"" complaints into concrete, measurable improvements backed by production data.
Scott Bastek is a Data Science professional at PagerDuty with a background from MIT. Based in Lisbon, he focuses on applying data science and modern AI techniques to solve practical engineering and operational challenges.
