A model can sound fluent, confident, and authoritative while being factually wrong, dangerously ungrounded, or silently broken. Recent failures, from AI-generated legal citations appearing before the England and Wales High Court to Air Canada being held liable after its chatbot gave incorrect bereavement fare information, show the same uncomfortable pattern: the output looked usable, but the evaluation layer failed. Most LLM failures in production are not only model failures. They are measurement failures. Teams swap prompts, upgrade models, and adjust retrieval logic without a systematic way of knowing what quietly got worse. This talk introduces a lightweight evaluation framework across six layers: correctness, grounding, format compliance, safety, refusal quality, and regression. Through a practical walkthrough of a lightweight 20-test evaluation harness, attendees will learn how to catch breakage before users do and treat evals like CI/CD for LLM behaviour.
Richard Dosumu is a cybersecurity practitioner, Founder at OctaTech, and an independent researcher focused on practical security in real-world environments. He holds an MSc in Cyber Security & Human Factors and writes educational cybersecurity content on Medium and LinkedIn, with a particular interest in making security guidance usable under operational pressure. His published research spans the intersection of AI, cybersecurity, and Industry 4.0/manufacturing, exploring how modern systems introduce new risks and control challenges. He is also involved with the OWASP Bristol organising community and is committed to sharing actionable, defensible approaches that teams can implement without disrupting critical operations.
