Your agent passes your benchmarks. The customer still won't ship it. Inside an enterprise you don't own, an eval isn't a leaderboard score; it's the artifact that convinces a skeptical stakeholder to put an agent in front of their users and their data. This talk walks through building evaluations an enterprise customer will actually believe: sourcing test cases from the customer's own failure history instead of a generic benchmark, making each failure legible enough that a non-engineer can see what broke and why, and turning "looked good in the demo" into a documented sign-off. Drawn from Forward Deployed work at Notion and an eval background from Scale AI, this covers the tradeoffs and the cases where this still falls short.
Paden Gayle is a Forward Deployed Engineer at Notion, previously a software engineer at Google and Bloomberg, and the author of an upcoming O'Reilly book on the Forward Deployed Engineer role. He started at Juilliard before finding his way into tech, and writes for ~23K people in tech on LinkedIn about breaking into Applied AI.
