LLMday is a worldwide series of community events for engineers building with large language models, AI agents and machine learning. Across cities around the world, we bring together practitioners working on AI-powered products and systems to share real-world experience, learn from each other and explore how software engineering is changing in the age of AI.
Our research introduces Visual AI Testing Agent, an agentic framework designed to revolutionize mobile UI automation for iOS Simulators by leveraging Vision Language Models (VLMs). Unlike traditional locator-based tools like Appium or XCUITest, which often fail due to unstable identifiers or dynamic UI changes, our VLM-driven approach treats the screen as the primary interface. The agent captures the screen, processes visible elements through vision-based providers, and executes actions with human-like visual understanding. This talk will demonstrate how this agentic architecture enables predictive test case generation and provides development teams with actionable, early-stage insights. By shifting to a vision-first paradigm, we enable more robust, resilient, and intelligent automated testing cycles.... Read more
The typical default is one call to one model, and when that gets expensive, we cascade to something cheaper. Both accept the same premise: the input is one question. Usually, it isn't. A single photo, document or utterance carries several distinct questions, and one model answering all of them hands you its weakest answer in its most confident voice, at full price, every time. A cascade only changes the price. This talk makes the case for treating the input as a routing problem. Cheap classifiers decide which question you have, and each question goes to the model genuinely best at it, including the times the honest answer is that no model should answer. In our case we brought token spend under control, took a seventeen second wait down to under a second on the path that can prove its answer, and cut wrong answers from 25% to near zero.... Read more
The Ogummaa Agent Portal is a comprehensive, sovereign AI workspace designed for secure, long-horizon task automation and conversational knowledge retrieval. The platform utilizes a decoupled microservices architecture—comprising an Admin Portal, Central Gateway, and Ingestion Pipeline—to ensure elastic scalability and rigorous environmental isolation. Its core engine, a GraphRAG (Graph Retrieval-Augmented Generation) and VectorRAG system for analytical graph processing, geo-spacial-temporal correlation, enabling sophisticated retrieval across massive enterprise datasets. The portal enforces a robust security paradigm through per-user sandbox isolation, a four-layer guardrail system, and a "bring your own secrets" model, while supporting unattended execution for recurring routines. By blending autonomous self-improvement loops with immutable event-based telemetry, and provides an enterprise-ready infrastructure for organizations prioritizing data sovereignty, multi-agent orchestration, and resilient AI deployment.... Read more
LLM systems are getting very good at extracting entities from messy documents. But extracting a mention is only the beginning. The harder problem is deciding what that mention actually refers to. A single real-world entity can appear under different names, formats, abbreviations, and contexts across documents and databases. At the same time, different entities can look remarkably similar. A reliable system therefore needs more than entity extraction or similarity search. It needs a mapping step that decides which candidate, if any, represents the entity being described. This is where similarity becomes both useful and insufficient. Database search, lexical matching, and embeddings are powerful ways to generate and rank candidates. But a high similarity score does not make a candidate the correct mapping. The system still has to account for ambiguity, context, available evidence, and what happens when no candidate meets the evidence threshold. Sometimes the right decision is not to map at all. This talk looks at entity mapping as a distinct engineering problem in an LLM pipeline: using structured search and semantic similarity to find plausible candidates, understanding where those signals break down, and designing the decision layer that determines when a candidate is good enough to map, and when the system should abstain. The goal is not to replace similarity with something else. It is to understand where similarity fits, and where the identity decision begins.... Read more
The most resilient companies will not bolt AI onto yesterday’s operating model—they will build with AI as part of the work from day one. This AI-native approach allows new companies to design roles, workflows, and teams intentionally, reducing the need for disruptive reactions when technology advances. The same principle applies to an unconventional career: candidates can use AI to research opportunities, identify decision-makers, track applications, prepare for interviews, and strengthen their day-to-day capabilities without outsourcing their judgment or agency. Drawing on more than a decade in recruiting and his experience building AstraPack, Nicholas will show how companies and candidates can develop their “AI muscle” early and remain competitive as the nature of work continues to change.... Read more
Most LLM evaluation assumes a gold answer exists to score against. A large and growing share of production work, however, doesn't have one: contract review, clinical documentation, incident summaries, moderation calls, research synthesis — any task where correctness is a matter of expert judgment, and where two qualified experts given the same input produce different outputs that are both defensible. If you score that as if a single right answer existed, you collapse the errors that matter — a fabricated fact, a dropped material detail — into the same number as the ones that don't, like structure, emphasis, and phrasing. This talk is about building evaluation for those systems, where the ground truth isn't a fixed key but a distribution of expert opinion, and where disagreement is built into the problem rather than a symptom of bad labeling. It unpacks three moves that improve accuracy and trustworthiness: treating inter-rater reliability among your experts as the ceiling on any eval you can build, using expert corrections as a live quality signal instead of a static gold set, and separating "genuinely wrong" from "differently right" so the score tracks the failures you actually care about.... Read more
No matter how well an AI pipeline is designed, it will always be limited by both quality and quantity of available data. However, real-world data rarely arrives nicely packaged and pre-normalized. In fact, it is probably safer to say that real-world data is the number one saboteur of existing machine learning applications. This talk will go over common data problems that have been seen in enterprise systems and the importance of curating data for pipelines of all varieties.... Read more
Self-managed AI model deployments on-prem and in private cloud infrastructure are becoming increasingly popular, accelerated by open-weight models that rival frontier models. But deploying them needs accelerators (such as GPUs) with large and expensive memory. The usual ways to save that memory quantize or approximate the model, thereby changing its output and affecting accuracy. This talk removes that tradeoff. It covers lossless compression of both model weights and the KV cache, across BF16, FP16, FP8, and FP4, delivering around 30% more GPU headroom with identical model output. Weights are bit-for-bit exact with cryptographic hash verification, and stay compressed in GPU memory while they run. Freeing weight memory leaves more room for the KV cache, and the KV cache is then compressed on top. In deployments that are already memory-constrained, this results in cases of up to 110% more KV cache headroom, raising usable context length, batch size, and concurrency. The talk frames bit-exact KV cache compression as the next lever for lossless inference efficiency.... Read more
This talk is a field report on using Grok Bot to sell commercial real estate. I’ll show the live stack I used on real deals: Grok Bot as the operating layer across buyer inbound and follow-up, Quo for the sales phone line and texts, a dedicated /buy sales website and deal room, Google Drive diligence packs, and offering memorandums, all powered by Grok Bot. We’ll cover what the agent actually does day to day (watching inbound, drafting first-touch that leads with income, logging buyer calls, keeping board copy and the OM in sync, chasing listing-agreement issues) and where humans still have to approve sends. Expect concrete CRE examples, the connectors and artifacts that made it work, and the failure modes that show up when an agent touches real buyers.... Read more
Autonomous research agents that propose, implement, and refine ML solutions have become remarkably capable. Yet most rely on frontier models throughout the research loop, creating substantial inference costs that limit their scalability and accessibility. This talk explores a resource-efficient alternative: a small open-weight model autonomously drives the research loop, while a frontier model serves as an on-demand advisor operating within a constrained inference budget. Drawing on principles of efficient AI inference and resource-aware AI engineering, we'll explore how inference-time decision-making, adaptive model routing, and dynamic compute allocation can balance model capability, inference cost, and research quality. We'll examine existing work on small-large model collaboration, agent distillation, and budget-aware inference, while addressing the challenges of long-horizon autonomous workflows, including unproductive exploration, error accumulation, and inefficient resource utilization. We'll introduce a framework for deciding when expensive model guidance is worth its cost, how much context the advisor needs, and how to measure its impact on research outcomes. A recorded demonstration will illustrate the autonomous research loop, intelligent advisor invocation, and inference budget utilization. The session concludes with open challenges in asynchronous model collaboration, persistent guidance, and engineering scalable, resource-efficient autonomous AI systems.... Read more
My wife, who doesn’t have any technical background, built a real app with AI, that she uses every day. So if someone with no previous experience can build working software now, what’s left for software engineers? During this presentation we'll explore what makes a software engineer relevant in the AI era. An engineer needs to know foundations, judgment, responsibility... Programming is easier. Software engineering isn’t.... Read more
15:30
Networking & sponsor crawl
Main lobby
16:00
Wrap up
Scan each other's QR codes & head to a nearby pub!
Visual AI Testing Agent: Screenshot-First Mobile UI Test Automation for iOS Simulator
Abstract
Our research introduces Visual AI Testing Agent, an agentic framework designed to revolutionize mobile UI automation for iOS Simulators by leveraging Vision Language Models (VLMs). Unlike traditional locator-based tools like Appium or XCUITest, which often fail due to unstable identifiers or dynamic UI changes, our VLM-driven approach treats the screen as the primary interface. The agent captures the screen, processes visible elements through vision-based providers, and executes actions with human-like visual understanding. This talk will demonstrate how this agentic architecture enables predictive test case generation and provides development teams with actionable, early-stage insights. By shifting to a vision-first paradigm, we enable more robust, resilient, and intelligent automated testing cycles.
Bio
Rahul Azmeera, PhD, is a Senior Software Engineer at 7-Eleven and an Independent Researcher specializing in AI and software engineering. He holds a PhD in Information Technology from the University of The Cumberlands and a Master's degree in Technology from Pittsburg State University.
With extensive hands-on experience in full-stack web application development, AWS cloud architecture, UX design, and QA, he specializes in mobile application development using hybrid frameworks like React Native and Ionic. His current research focuses on vision-language models for intelligent software testing, agentic AI, and analyzing the root causes of software errors.
Geoff Niehaus
Today Mechanic
One LLM Call Is the Wrong Default
Abstract
The typical default is one call to one model, and when that gets expensive, we cascade to something cheaper. Both accept the same premise: the input is one question. Usually, it isn't. A single photo, document or utterance carries several distinct questions, and one model answering all of them hands you its weakest answer in its most confident voice, at full price, every time. A cascade only changes the price. This talk makes the case for treating the input as a routing problem. Cheap classifiers decide which question you have, and each question goes to the model genuinely best at it, including the times the honest answer is that no model should answer. In our case we brought token spend under control, took a seventeen second wait down to under a second on the path that can prove its answer, and cut wrong answers from 25% to near zero.
Bio
Geoff Niehaus is the founder of Today Mechanic, an AI-powered platform that keeps auto repair shops' bays full by handling the customer conversation before the car ever reaches the lift. He spent the last four years at CVS Health (via Oak Street Health), most recently as Executive Director of Software Development Engineering, where he oversaw a program of 100+ engineers and led delivery of a production applied-AI system (retrieval over millions of clinical documents feeding an LLM) that lifted suspect-finding identification by 30% under HIPAA constraints and human-in-the-loop clinical validation. He also embedded AI-assisted development into the standard SDLC, pairing coding agents with a contract-first "Design First" practice that took code coverage from 40% to 80%+. He's a founding member of the Austin CTO Club and holds a pending patent for multi-tenant medical record access.
Rodney Richard Puplampu
Puplampu Consulting
Ogummaa Agent Portal. Architecture & Security Blueprint for Long-Horizon Agents
Abstract
The Ogummaa Agent Portal is a comprehensive, sovereign AI workspace designed for secure, long-horizon task automation and conversational knowledge retrieval. The platform utilizes a decoupled microservices architecture—comprising an Admin Portal, Central Gateway, and Ingestion Pipeline—to ensure elastic scalability and rigorous environmental isolation. Its core engine, a GraphRAG (Graph Retrieval-Augmented Generation) and VectorRAG system for analytical graph processing, geo-spacial-temporal correlation, enabling sophisticated retrieval across massive enterprise datasets. The portal enforces a robust security paradigm through per-user sandbox isolation, a four-layer guardrail system, and a "bring your own secrets" model, while supporting unattended execution for recurring routines. By blending autonomous self-improvement loops with immutable event-based telemetry, and provides an enterprise-ready infrastructure for organizations prioritizing data sovereignty, multi-agent orchestration, and resilient AI deployment.
Bio
Rodney Puplampu, Principal AI Platform Engineer, Puplampu Consulting, author of What Everyone Should Know About the Rise of AI, creator of the Ogummaa Sovereign AI Platform, born in Detroit, Michigan in 1974, is a passionate advocate for safe, transparent, and ethical Artificial Intelligence (AI). With over thirty years of experience in network, security, telecom, full-stack applications, and sovereign agentic AI platforms, Rodney has held positions in the United States Army and worked for renowned companies such as AT&T, Lucent Technologies, Avaya, and HP. Throughout his career, he has consistently demonstrated his commitment to ensuring responsible and ethical business practices.
Rodney also has a passion for cleaning the environment. This extends to his advocacy for AI because he believes AI has the potential to help mankind address global challenges such as climate change and sustainable resource management.
His dedication to safety and transparency in AI is evident in his volunteer work and service to countless causes. He strongly believes that AI should be designed with safeguards to prevent it from being used for malicious purposes or to perpetuate discrimination. Rodney recognizes that AI should complement and enhance human capabilities rather than replace them.
Maryam Astero
Genomenon
One Thing, Many Names: Building Reliable Entity Mapping
Abstract
LLM systems are getting very good at extracting entities from messy documents. But extracting a mention is only the beginning. The harder problem is deciding what that mention actually refers to.
A single real-world entity can appear under different names, formats, abbreviations, and contexts across documents and databases. At the same time, different entities can look remarkably similar. A reliable system therefore needs more than entity extraction or similarity search. It needs a mapping step that decides which candidate, if any, represents the entity being described.
This is where similarity becomes both useful and insufficient. Database search, lexical matching, and embeddings are powerful ways to generate and rank candidates. But a high similarity score does not make a candidate the correct mapping. The system still has to account for ambiguity, context, available evidence, and what happens when no candidate meets the evidence threshold. Sometimes the right decision is not to map at all.
This talk looks at entity mapping as a distinct engineering problem in an LLM pipeline: using structured search and semantic similarity to find plausible candidates, understanding where those signals break down, and designing the decision layer that determines when a candidate is good enough to map, and when the system should abstain.
The goal is not to replace similarity with something else. It is to understand where similarity fits, and where the identity decision begins.
Bio
Maryam Astero is a Senior AI Research Engineer at Genomenon, where she builds production systems that turn unstructured scientific literature into structured, queryable knowledge. Her work spans document ranking, information extraction, entity resolution, and knowledge graph construction, with a focus on making these systems reliable beyond the prototype stage.
She holds a PhD in Computer Science from Aalto University, where she developed graph neural network architectures for chemical reaction modeling. She works at the meeting point of machine learning, graph-based methods, and production systems, particularly where ambiguity and imperfect data make seemingly simple problems hard to evaluate and scale.
Nick Smith
AstraPack
AI-Native, Not AI-Reactive: Rethinking How We Build Companies and Careers
Abstract
The most resilient companies will not bolt AI onto yesterday’s operating model—they will build with AI as part of the work from day one. This AI-native approach allows new companies to design roles, workflows, and teams intentionally, reducing the need for disruptive reactions when technology advances. The same principle applies to an unconventional career: candidates can use AI to research opportunities, identify decision-makers, track applications, prepare for interviews, and strengthen their day-to-day capabilities without outsourcing their judgment or agency. Drawing on more than a decade in recruiting and his experience building AstraPack, Nicholas will show how companies and candidates can develop their “AI muscle” early and remain competitive as the nature of work continues to change.
Bio
Nick is the Co-founder and Head of Talent at AstraPack, a boutique recruiting and career-strategy firm. Over the past decade, he has built talent systems and recruited inside MAANG, EV, and other demanding technology organizations before founding AstraPack to bring that experience directly to growing companies and professionals navigating career transitions. He shares practical perspectives on hiring, AI, and career strategy for new grads and tenured professionals alike with a LinkedIn audience of nearly 25,000 people.
Leila Anderson
Upheal
Evaluation Without Ground Truth: Lessons from Expert Disagreement
Abstract
Most LLM evaluation assumes a gold answer exists to score against. A large and growing share of production work, however, doesn't have one: contract review, clinical documentation, incident summaries, moderation calls, research synthesis — any task where correctness is a matter of expert judgment, and where two qualified experts given the same input produce different outputs that are both defensible. If you score that as if a single right answer existed, you collapse the errors that matter — a fabricated fact, a dropped material detail — into the same number as the ones that don't, like structure, emphasis, and phrasing.
This talk is about building evaluation for those systems, where the ground truth isn't a fixed key but a distribution of expert opinion, and where disagreement is built into the problem rather than a symptom of bad labeling. It unpacks three moves that improve accuracy and trustworthiness: treating inter-rater reliability among your experts as the ceiling on any eval you can build, using expert corrections as a live quality signal instead of a static gold set, and separating "genuinely wrong" from "differently right" so the score tracks the failures you actually care about.
Bio
Leila Anderson is a Senior Clinical AI Product Manager at Upheal, where she builds and evaluates the LLM systems behind clinical documentation for behavioral health. She's also a licensed marriage and family therapist (LMFT-S) with nearly a decade of clinical experience, which puts her on both sides of the eval problem this talk is about: the expert whose judgment the model is trying to match, and the person on the hook for deciding whether it did. Alongside her work at Upheal she runs a private psychotherapy practice and serves as an expert witness in clinical cases — two more roles where being the arbiter of a contested-but-defensible judgment is the whole job.
Colby Mainard
RiskScout
Invisible Data - The Largest Problem Facing AI
Abstract
No matter how well an AI pipeline is designed, it will always be limited by both quality and quantity of available data. However, real-world data rarely arrives nicely packaged and pre-normalized. In fact, it is probably safer to say that real-world data is the number one saboteur of existing machine learning applications. This talk will go over common data problems that have been seen in enterprise systems and the importance of curating data for pipelines of all varieties.
Bio
Colby Mainard is a machine learning engineer with more than five years of experience designing, deploying, and operating production ML systems, currently working as an AI/ML engineer at RiskScout on transaction anomaly detection, entity risk scoring, and network analysis across financial data. He holds a Master of Computer Science from Texas A&M University with a focus on artificial intelligence, machine learning, deep learning, and computer vision, along with a B.S. in Computer Science from the same institution with minors in business and cybersecurity. Earlier roles span sports analytics and cybersecurity, including computer vision pipelines for the NFL, NHL, MLB, and NBA at MVP, as well as Python and C++ data tooling for threat detection at Vectra. Outside of work he studies quantum computing, plays Dungeons and Dragons, enjoys learning about history, and shoots landscape photography.
Kunle Olutomilayo
Isiro AI
30% More KV Cache Headroom, 100% Accuracy
Abstract
Self-managed AI model deployments on-prem and in private cloud infrastructure are becoming increasingly popular, accelerated by open-weight models that rival frontier models. But deploying them needs accelerators (such as GPUs) with large and expensive memory. The usual ways to save that memory quantize or approximate the model, thereby changing its output and affecting accuracy.
This talk removes that tradeoff. It covers lossless compression of both model weights and the KV cache, across BF16, FP16, FP8, and FP4, delivering around 30% more GPU headroom with identical model output. Weights are bit-for-bit exact with cryptographic hash verification, and stay compressed in GPU memory while they run. Freeing weight memory leaves more room for the KV cache, and the KV cache is then compressed on top. In deployments that are already memory-constrained, this results in cases of up to 110% more KV cache headroom, raising usable context length, batch size, and concurrency. The talk frames bit-exact KV cache compression as the next lever for lossless inference efficiency.
Bio
Kunle Olutomilayo is the Founder of Isiro AI, lowering the cost of ownership for self-managed AI infrastructure. The company focuses on lossless reduction of the hardware memory required for model deployments, without changing the model output. Kunle's core thesis is that the next wave of inference cost savings will not only come from faster chips or smaller models, but also from kernel software breakthroughs that move less data to address the memory wall problem, without changing what the model produces. This is especially important in regulated and high-stakes environments, such as finance, health care, defense, and enterprise AI, where cost savings must not come at the expense of predictable model behavior. Kunle holds a PhD in Electrical Engineering and previously worked on advanced AI and vehicle autonomy at Ford Greenfield Labs.
Matt Barge
Superbuilders
Selling Commercial Real Estate with Grok Bot
Abstract
This talk is a field report on using Grok Bot to sell commercial real estate. I’ll show the live stack I used on real deals: Grok Bot as the operating layer across buyer inbound and follow-up, Quo for the sales phone line and texts, a dedicated /buy sales website and deal room, Google Drive diligence packs, and offering memorandums, all powered by Grok Bot.
We’ll cover what the agent actually does day to day (watching inbound, drafting first-touch that leads with income, logging buyer calls, keeping board copy and the OM in sync, chasing listing-agreement issues) and where humans still have to approve sends. Expect concrete CRE examples, the connectors and artifacts that made it work, and the failure modes that show up when an agent touches real buyers.
Bio
Matt Barge is a Forward Deployed AI Engineer at Superbuilders, based in Austin, Texas. He builds production AI systems for go-to-market and operations, and manages and sells commercial real estate assets in Texas. He previously built iOS products at MoneyGram and has spent the last several years putting LLM agents to work. More at mattbarge.com.
Vashishtha Patil
Amazon Lab126
Auto-Research on a Budget: Small Models in the Loop, Frontier Models on Call
Abstract
Autonomous research agents that propose, implement, and refine ML solutions have become remarkably capable. Yet most rely on frontier models throughout the research loop, creating substantial inference costs that limit their scalability and accessibility.
This talk explores a resource-efficient alternative: a small open-weight model autonomously drives the research loop, while a frontier model serves as an on-demand advisor operating within a constrained inference budget.
Drawing on principles of efficient AI inference and resource-aware AI engineering, we'll explore how inference-time decision-making, adaptive model routing, and dynamic compute allocation can balance model capability, inference cost, and research quality. We'll examine existing work on small-large model collaboration, agent distillation, and budget-aware inference, while addressing the challenges of long-horizon autonomous workflows, including unproductive exploration, error accumulation, and inefficient resource utilization.
We'll introduce a framework for deciding when expensive model guidance is worth its cost, how much context the advisor needs, and how to measure its impact on research outcomes. A recorded demonstration will illustrate the autonomous research loop, intelligent advisor invocation, and inference budget utilization. The session concludes with open challenges in asynchronous model collaboration, persistent guidance, and engineering scalable, resource-efficient autonomous AI systems.
Bio
Vashishtha Patil is a Senior Applied Scientist at Amazon Lab126, specializing in LLM-powered AI for the Alexa+ Smart Home experience. With over a decade of experience across Amazon and Qualcomm, his expertise spans machine learning, computer vision, and algorithm development for smart home, healthtech, and mobile devices. He holds a Master's degree from the University of Southern California.
Nate Lemos
MakingFriends.app
My wife is a vibe-coder. So, what now?
Abstract
My wife, who doesn’t have any technical background, built a real app with AI, that she uses every day. So if someone with no previous experience can build working software now, what’s left for software engineers?
During this presentation we'll explore what makes a software engineer relevant in the AI era. An engineer needs to know foundations, judgment, responsibility...
Programming is easier. Software engineering isn’t.
Bio
Nate Lemos is the co-founder of MakingFriends.app and an experienced engineer with over 12 years building systems for all sizes of business. Nate was a key contributor of the Halo Infinite backend and migration of the Azure Portal to Typescript. Now, Nate writes enterprise AI agents for thousands of users at ECI and runs his own app to help people meet IRL.