LLMday is a worldwide series of community events for engineers building with large language models, AI agents and machine learning. Across cities around the world, we bring together practitioners working on AI-powered products and systems to share real-world experience, learn from each other and explore how software engineering is changing in the age of AI.
Companies presenting:
AWS, Datadog, Dell Technologies, IBM, KodeKloud, Microsoft, Red Hat, The Austin Consulting Group
AI keeps getting sharper, faster, and more embedded in how we build and run systems. It can summarize incidents, suggest fixes, and even take action. Yet when things break at 3 a.m., the pager still lands on a human.
This keynote explores the gap between impressive AI capability and real operational trust. We will look at where agentic systems are already delivering value, where they quietly fall apart, and why reliability, context, and accountability remain hard problems. The promise of fully autonomous operations is getting closer, but we are not off the hook yet.... Read more
What happens when AI agents encounter tasks they cannot solve? Most agents fail because their capabilities are fixed. This talk explores a new approach using only open-source frameworks where agents can now adapt themselves by dynamically generating and integrating new tools at runtime. We will examine the architectural mechanisms that enable this, including how new functionality is validated and incorporated safely during execution. Through a live demo, attendees will see an agent encounter a novel task, build the required capability, and complete the workflow without redeployment. Attendees will additionally gain practical engineering insights into the tradeoffs and design considerations involved in building autonomous agentic systems that evolve beyond their original design.... Read more
Every team says they want better observability, but adding OpenTelemetry instrumentation is repetitive, tedious, convention-heavy work. Wah wah wahhhh, what a drag. Enter Spinybacked Orbweaver to save the day! Spinybacked Orbweaver is an AI-powered telemetry agent that can analyze your code, add the instrumentation, check its own work, and open a pull request. And it can do it consistently well, following your company's telemetry standards! This speaker built it, and she'll demo it live. Sparkles and rainbows! But here's where it gets interesting: how does the agent know if it did a good job? Two things. First, the OpenTelemetry project has a tool called Weaver that lets you define your team's telemetry standards as a machine-readable schema. Second, there's an emerging open specification called the Instrumentation Score that rates instrumentation quality on a 0-to-100 scale using standardized rules. The agent validates against both of these, plus some custom rules. When the agent fails its own checks, it gets the feedback and tries again. Generate, validate, learn, retry. You'll watch the whole thing happen against a real codebase. The agent discovers what to instrument, writes the code, gets rejected by its own quality checks, fixes its mistakes, and produces a PR. Along the way, you'll get grounded in what OpenTelemetry, Weaver, and the Instrumentation Score are, why they exist, and why good telemetry is worth caring about... but not enough to suffer the tedium of manual code instrumentation.... Read more
As AI moves from experimental pilots to production MLOps pipelines, defining what truly constitutes **"Open Source AI"** has become a critical engineering challenge. This session cuts through the hype to provide ML engineers, data scientists, and infrastructure architects with practical frameworks to evaluate model transparency, mitigate licensing risks, and navigate the complexities of deploying "open" foundation models. Attendees will walk away with the skills to: - **Deconstruct the AI Stack**: Understand the core differences between traditional open-source software and the AI triad. - **Evaluate True Openness**: Apply frameworks like the Open Source AI Definition (OSAID) to assess model viability for fine-tuning. - **Spot "Openwashing"**: Identify the operational risks and licensing traps hidden behind "open weights" models to protect your deployment pipelines and long-term infrastructure.... Read more
Most enterprise AI never makes it past the pilot phase. The systems that do survive production at Fortune 50 scale — across hundreds of thousands of daily transactions, thousands of human operators, and dozens of countries — share architectural patterns that aren’t obvious at the demo stage. This talk distills those patterns from two production AI programs built and deployed over the past decade: a digital supply-chain transformation platform supporting 220,000 orders per day across 392,000 SKUs, documented as MIT Sloan / Ivey case study W24797 (a Harvard Business Publishing bestseller, taught at MIT, Rutgers, NUS, and IE Business School); and STRIDE, a closed-loop AI service architecture supporting customer support organization, recognized with the 2025 TSIA STAR Award for Innovation in Analytics and AI for Service Excellence. Both programs faced the same failure modes that LLM teams hit today: hallucinated outputs, brittle integration boundaries, runaway cost curves, and slow learning loops. Both succeeded by converging on the same underlying architecture — continuous telemetry feeding decision engines feeding learning loops, with humans gating the boundaries rather than sitting in the critical path. The talk walks through the architectural decisions that determined whether each system shipped or stalled, including build-vs-buy under vendor immaturity, phased rollout under quality risk, and tiered automation under fraud exposure. ... Read more
Most LLM training data and enterprise RAG pipelines depend on document ingestion, yet traditional tools waste tokens on lossy formats and struggle with real-world PDFs. The cost compounds at scale: inefficient document representation means higher training costs, larger context windows consumed, and degraded retrieval quality.
In this talk, I'll introduce Docling, an open-source document processing engine that takes a different approach. Docling uses purpose-built deep learning models to parse documents the way humans read them, preserving hierarchy, extracting tables, and maintaining reading order. I'll also introduce DocTags, a markup language we designed specifically for LLM tokenizers. DocTags uses 30-45% fewer tokens than HTML for the same content, and each tag maps directly to a single token, so you're not wasting context window space on markup overhead.
We'll look at how IBM and Hugging Face used Docling to process 475 million PDFs for the FinePDFs dataset, extracting 3 trillion tokens at 50x lower cost than VLM-based approaches. I'll also cover IBM's work through LF AI & Data to establish DocTags as an ISO standard, creating a shared representation format that's optimized for both LLM training and inference.
Docling is fully open-source under MIT license, so you can run it locally for sensitive data and air-gapped environments.
... Read more
In 2025 I gave an AI agent cluster-level permissions and walked away. Forty minutes later I had no cluster. The agent didn't hallucinate. It didn't go rogue. It force-refreshed etcd, then while trying to fix that, wiped the netplan configuration on every Linux node in the cluster. One bad decision made things worse. No gate stopped it anywhere in the chain. The instinct after an incident like that is to put humans back in the loop. That's the wrong lesson. The whole point of agentic AI is autonomous operation at machine speed. Slowing it down with manual approval gates defeats the purpose. The right fix is deterministic programmatic gates that let the agent move fast within a defined safe boundary. Most operations should never require a human. A deletion that meets the right criteria gets approved automatically. A deletion that doesn't gets blocked automatically. The safety guarantee comes from the gate logic, not from someone watching the terminal. This talk walks through the full failure chain and the Eight Guardrails Framework that came out of it: three enforcement layers spanning Claude Code pre-tool-use hooks, Git hooks, and Kubernetes infrastructure controls. These are not probabilistic guardrails. They are deterministic checks that fire before execution, enforce IaC-only infrastructure changes, apply least privilege so a dev-context agent can't reach control plane nodes, and trigger automated rollback when something goes wrong. The agent operates autonomously inside that boundary. Outside it, the pipeline stops it cold. The core principle: don't use probabilistic AI to enforce deterministic requirements. Build the gates programmatically, test them the same way you test your code, and let the agent run. Attendees will leave with the exact failure chain mapped to the gate that would have stopped it, the Eight Guardrails applied to concrete controls across their pipeline, and a gap checklist to find where their own setup lets agents operate outside a safe boundary.... Read more
Large language models have moved rapidly from experimentation to production, yet many real world systems fail to deliver reliable outcomes. From hallucinations and inconsistent responses to latency spikes and escalating costs, the gap between demo and production remains significant. This talk explores why most LLM systems break in production and how to design them differently. Drawing from hands on experience building large scale user facing systems, the session highlights common failure modes and the architectural decisions that address them. We will dive into practical patterns such as retrieval augmented generation (RAG), prompt orchestration, evaluation pipelines, and cost aware design. The talk will also cover real trade offs such as accuracy versus latency, flexibility versus control, and innovation versus reliability. Attendees will leave with a clear framework for building LLM systems that are not just impressive in demos but dependable in production.... Read more
Most enterprise AI demos start with a single assistant or copilot. However, real business workflows require orchestration, validation, approval, monitoring, and integration with existing systems. This talk explains how to design multi-agent workflows with human-in-the-loop control so LLM applications can move from demos to production.... Read more
LLMs are useful as copilots, but the real shift begins when they coordinate work across people, systems, and decisions. This talk shows how organizations design agentic workflows with guardrails, escalation paths, and real operational value.... Read more
Modern enterprises face a critical paradox: despite processing billions of records and building sophisticated data infrastructure, business users often remain trapped in manual spreadsheets and wait days for simple insights. I solved this challenge by implementing agentic AI on top of an AWS data pipeline processing 45 billion records per month, enabling over 2,000 users across 37 teams to perform autonomous natural language analysis. This implementation achieved 25-40% productivity gains and a 60% reduction in time-to-insight. This talk explores the evolution from traditional BI dashboards to AI copilots and fully autonomous agents, demonstrating how agentic AI transforms data accessibility and shifts data teams from report factories to strategic partners.... Read more
17:30
Wrap up
Scan each other's QR codes & head to a nearby pub!
What happens when AI agents encounter tasks they cannot solve? Most agents fail because their capabilities are fixed. This talk explores a new approach using only open-source frameworks where agents can now adapt themselves by dynamically generating and integrating new tools at runtime. We will examine the architectural mechanisms that enable this, including how new functionality is validated and incorporated safely during execution. Through a live demo, attendees will see an agent encounter a novel task, build the required capability, and complete the workflow without redeployment. Attendees will additionally gain practical engineering insights into the tradeoffs and design considerations involved in building autonomous agentic systems that evolve beyond their original design.
Bio
Sandhya Subramani is a Senior Developer Advocate for Generative AI at Amazon Web Services, specializing in applied AI research and production-grade machine learning systems. She previously served as a Lead Applied AI Scientist at a stealth AI startup and as a Data Scientist at Warner Bros. Discovery, where she focused on applied AI/ML research and data-driven product development. Earlier in her career, she worked as an AI Engineer at Fidelity Investments within enterprise cybersecurity. Sandhya’s expertise spans generative AI, NLP, applied machine learning research, and translating advanced AI capabilities into real-world applications.
Whitney Lee
Datadog
The Best Laid Spans: An AI Agent That Instruments Your Code with OpenTelemetry
Abstract
Every team says they want better observability, but adding OpenTelemetry instrumentation is repetitive, tedious, convention-heavy work. Wah wah wahhhh, what a drag. Enter Spinybacked Orbweaver to save the day! Spinybacked Orbweaver is an AI-powered telemetry agent that can analyze your code, add the instrumentation, check its own work, and open a pull request. And it can do it consistently well, following your company's telemetry standards! This speaker built it, and she'll demo it live. Sparkles and rainbows! But here's where it gets interesting: how does the agent know if it did a good job? Two things. First, the OpenTelemetry project has a tool called Weaver that lets you define your team's telemetry standards as a machine-readable schema. Second, there's an emerging open specification called the Instrumentation Score that rates instrumentation quality on a 0-to-100 scale using standardized rules. The agent validates against both of these, plus some custom rules. When the agent fails its own checks, it gets the feedback and tries again. Generate, validate, learn, retry. You'll watch the whole thing happen against a real codebase. The agent discovers what to instrument, writes the code, gets rejected by its own quality checks, fixes its mistakes, and produces a PR. Along the way, you'll get grounded in what OpenTelemetry, Weaver, and the Instrumentation Score are, why they exist, and why good telemetry is worth caring about... but not enough to suffer the tedium of manual code instrumentation.
Bio
Whitney Lee is a creator and systems thinker who explores how observability, AI, and platform engineering connect across the cloud native ecosystem. She brings humor, depth, and clarity to complex technologies while building original frameworks that help others understand how systems fit together. She runs a vibrant YouTube channel, hosts Datadog Illuminated and Software Defined Interviews, has delivered two KubeCon keynotes and countless breakout talks, and combines storytelling and technical rigor to illuminate the human side of cloud native engineering.
Alfonso Cancellara
Red Hat
Beyond the Hype: Navigating "Openwashing" and the Real Open Source AI
Abstract
As AI moves from experimental pilots to production MLOps pipelines, defining what truly constitutes "Open Source AI" has become a critical engineering challenge. This session cuts through the hype to provide ML engineers, data scientists, and infrastructure architects with practical frameworks to evaluate model transparency, mitigate licensing risks, and navigate the complexities of deploying "open" foundation models. Attendees will walk away with the skills to: - Deconstruct the AI Stack: Understand the core differences between traditional open-source software and the AI triad. - Evaluate True Openness: Apply frameworks like the Open Source AI Definition (OSAID) to assess model viability for fine-tuning. - Spot "Openwashing": Identify the operational risks and licensing traps hidden behind "open weights" models to protect your deployment pipelines and long-term infrastructure.
Bio
Alfonso Cancellara is an OpenShift-focused Red Hat Technical Account Manager. Beyond his day-to-day role, he's a passionate Open Source advocate and a speaker at conferences and industry events, exploring how Open Source Software fuels innovation.
Ankit Talwar
Dell Technologies
Production AI at Enterprise Scale: Lessons from Two Fortune 50 Deployments
Abstract
Most enterprise AI never makes it past the pilot phase. The systems that do survive production at Fortune 50 scale — across hundreds of thousands of daily transactions, thousands of human operators, and dozens of countries — share architectural patterns that aren’t obvious at the demo stage. This talk distills those patterns from two production AI programs built and deployed over the past decade: a digital supply-chain transformation platform supporting 220,000 orders per day across 392,000 SKUs, documented as MIT Sloan / Ivey case study W24797 (a Harvard Business Publishing bestseller, taught at MIT, Rutgers, NUS, and IE Business School); and STRIDE, a closed-loop AI service architecture supporting customer support organization, recognized with the 2025 TSIA STAR Award for Innovation in Analytics and AI for Service Excellence. Both programs faced the same failure modes that LLM teams hit today: hallucinated outputs, brittle integration boundaries, runaway cost curves, and slow learning loops. Both succeeded by converging on the same underlying architecture — continuous telemetry feeding decision engines feeding learning loops, with humans gating the boundaries rather than sitting in the critical path. The talk walks through the architectural decisions that determined whether each system shipped or stalled, including build-vs-buy under vendor immaturity, phased rollout under quality risk, and tiered automation under fraud exposure.
Bio
Ankit Talwar is Director of Product Management for AI at Dell Technologies and a Distinguished Fellow of the Soft Computing Research Society (SCRS). He led the product strategy for the digital supply chain transformation documented by researchers at the Massachusetts Institute of Technology (MIT Digital Supply Chain case study, “Dell: Roadmap of a Digital Supply Chain Transformation,” Case No. W24797), and directed the STRIDE service AI ecosystem recognized with the 2025 TSIA STAR Award for Innovation in Analytics and AI for Service Excellence. His work synthesizes patterns from these programs across eight years (2017–2025). Mr. Talwar serves as a judge for MIT Hack-Nation’s Global AI Hackathon (three consecutive editions and the Venture Lab) and as a proposal reviewer for Manning Publications on agentic AI titles.
Mingxuan Zhao
IBM
Unlocking Document Intelligence with Open-Source AI
Abstract
Most LLM training data and enterprise RAG pipelines depend on document ingestion, yet traditional tools waste tokens on lossy formats and struggle with real-world PDFs. The cost compounds at scale: inefficient document representation means higher training costs, larger context windows consumed, and degraded retrieval quality.
In this talk, I'll introduce Docling, an open-source document processing engine that takes a different approach. Docling uses purpose-built deep learning models to parse documents the way humans read them, preserving hierarchy, extracting tables, and maintaining reading order. I'll also introduce DocTags, a markup language we designed specifically for LLM tokenizers. DocTags uses 30-45% fewer tokens than HTML for the same content, and each tag maps directly to a single token, so you're not wasting context window space on markup overhead.
We'll look at how IBM and Hugging Face used Docling to process 475 million PDFs for the FinePDFs dataset, extracting 3 trillion tokens at 50x lower cost than VLM-based approaches. I'll also cover IBM's work through LF AI & Data to establish DocTags as an ISO standard, creating a shared representation format that's optimized for both LLM training and inference.
Docling is fully open-source under MIT license, so you can run it locally for sensitive data and air-gapped environments.
Bio
Mingxuan Zhao is a Software Developer II at IBM working in Open Technology. A graduate of the Cockrell School of Engineering at the University of Texas at Austin, Mingxuan focuses on building developer-focused tools and contributing to modern web and cloud technologies. Based in New York, Mingxuan works on open and collaborative engineering initiatives within IBM.
Michael Forrester
KodeKloud
Your MLOps Pipeline is your Agentic AI Guardrail
Abstract
In 2025 I gave an AI agent cluster-level permissions and walked away. Forty minutes later I had no cluster. The agent didn't hallucinate. It didn't go rogue. It force-refreshed etcd, then while trying to fix that, wiped the netplan configuration on every Linux node in the cluster. One bad decision made things worse. No gate stopped it anywhere in the chain. The instinct after an incident like that is to put humans back in the loop. That's the wrong lesson. The whole point of agentic AI is autonomous operation at machine speed. Slowing it down with manual approval gates defeats the purpose. The right fix is deterministic programmatic gates that let the agent move fast within a defined safe boundary. Most operations should never require a human. A deletion that meets the right criteria gets approved automatically. A deletion that doesn't gets blocked automatically. The safety guarantee comes from the gate logic, not from someone watching the terminal. This talk walks through the full failure chain and the Eight Guardrails Framework that came out of it: three enforcement layers spanning Claude Code pre-tool-use hooks, Git hooks, and Kubernetes infrastructure controls. These are not probabilistic guardrails. They are deterministic checks that fire before execution, enforce IaC-only infrastructure changes, apply least privilege so a dev-context agent can't reach control plane nodes, and trigger automated rollback when something goes wrong. The agent operates autonomously inside that boundary. Outside it, the pipeline stops it cold. The core principle: don't use probabilistic AI to enforce deterministic requirements. Build the gates programmatically, test them the same way you test your code, and let the agent run. Attendees will leave with the exact failure chain mapped to the gate that would have stopped it, the Eight Guardrails applied to concrete controls across their pipeline, and a gap checklist to find where their own setup lets agents operate outside a safe boundary.
Bio
Principal Training Architect with 30 years of infrastructure experience across federal, Fortune 50, and startup environments. Has personally taught over 100,000 engineers platform engineering and AI/ML infrastructure. Workshop ranked third most-selected at KCD Texas 2026. Speaker at KubeCon EU Cloud Native University and KubeAuto AI Day Europe, Amsterdam. The cluster deletion incident described in this talk actually happened.
Varun Joshi
AWS
Why Most LLM Systems Fail in Production and How to Fix Them
Abstract
Large language models have moved rapidly from experimentation to production, yet many real world systems fail to deliver reliable outcomes. From hallucinations and inconsistent responses to latency spikes and escalating costs, the gap between demo and production remains significant. This talk explores why most LLM systems break in production and how to design them differently. Drawing from hands on experience building large scale user facing systems, the session highlights common failure modes and the architectural decisions that address them. We will dive into practical patterns such as retrieval augmented generation (RAG), prompt orchestration, evaluation pipelines, and cost aware design. The talk will also cover real trade offs such as accuracy versus latency, flexibility versus control, and innovation versus reliability. Attendees will leave with a clear framework for building LLM systems that are not just impressive in demos but dependable in production.
Bio
Varun Joshi is a Senior Data Engineer at Amazon Web Services (AWS) based in the Greater Seattle Area, with over 12 years of experience in data integration, data modeling, optimization, and engineering. He is an alumnus of the University of South Florida and has built a strong professional network with 4,500+ followers on LinkedIn.
Farhan Shahnewaz
Microsoft
Multi-Agent Workflows with Human-in-the-Loop Control
Abstract
Most enterprise AI demos start with a single assistant or copilot. However, real business workflows require orchestration, validation, approval, monitoring, and integration with existing systems. This talk explains how to design multi-agent workflows with human-in-the-loop control so LLM applications can move from demos to production.
Bio
Farhan Shahnewaz is an AI & Cloud Solutions Engineer at Microsoft with over 20+ years of experience across AI, Azure, cloud architecture, and enterprise solution design. He has worked in principal cloud architect roles, also in software engineering and research positions. Farhan is also the author of SMART NPC: AI for Game Character.
Jim Markunas
The Austin Consulting Group
From Copilots to Agents: Designing LLM Workflows for Real Organizations
Abstract
LLMs are useful as copilots, but the real shift begins when they coordinate work across people, systems, and decisions. This talk shows how organizations design agentic workflows with guardrails, escalation paths, and real operational value.
Bio
Jim Markunas is a Senior Technical Program and Project Manager and Technical Product Owner who specializes in helping enterprise teams deliver reliable eCommerce and SDLC releases. With experience at organizations like DIRECTV, CPS Energy, and New York Life, he leads complex platform modernization and integration initiatives, bringing structure and clarity to teams facing delivery challenges. Known for his data-driven approach, he focuses on restoring stakeholder confidence and ensuring predictable, high-quality product launches.
Srinivas Pochincharla
AWS
From Automation to Autonomy: Building Product-Led Agentic AI Systems That Scale
Abstract
Modern enterprises face a critical paradox: despite processing billions of records and building sophisticated data infrastructure, business users often remain trapped in manual spreadsheets and wait days for simple insights. I solved this challenge by implementing agentic AI on top of an AWS data pipeline processing 45 billion records per month, enabling over 2,000 users across 37 teams to perform autonomous natural language analysis. This implementation achieved 25-40% productivity gains and a 60% reduction in time-to-insight. This talk explores the evolution from traditional BI dashboards to AI copilots and fully autonomous agents, demonstrating how agentic AI transforms data accessibility and shifts data teams from report factories to strategic partners.
Bio
Srinivas Pochincharla is a Senior Technical Program Manager at AWS, focused on building and scaling large enterprise data platforms. He specializes in data architecture and agile program delivery, leading initiatives like high-volume data pipelines that process billions of records to support cost allocation and forecasting.
Miko Pawlikowski
Tech Author
KeynoteI Was Promised a Beach: Why AI Hasn’t Taken Your Pager (Yet)
Abstract
AI keeps getting sharper, faster, and more embedded in how we build and run systems. It can summarize incidents, suggest fixes, and even take action. Yet when things break at 3 a.m., the pager still lands on a human.
This keynote explores the gap between impressive AI capability and real operational trust. We will look at where agentic systems are already delivering value, where they quietly fall apart, and why reliability, context, and accountability remain hard problems. The promise of fully autonomous operations is getting closer, but we are not off the hook yet.
Bio
Miko Pawlikowski is an SRE Author and a platform engineer at Quadrature. He has led large-scale infrastructure and SRE initiatives at Citadel and Bloomberg, with deep expertise in Kubernetes, cloud computing, and chaos engineering. Passionate about building resilient systems and communities, he brings together engineers worldwide through conferences and media projects.
