LLMday

Large Language Models, Agents & AI Systems

October 29, 2026 Microsoft Experience Center NYC, United States

1
Day
10+
Speakers
1
Track
100+
Attendees

What Did Your Agent Actually Do?

Maureese Williams
Disney
Abstract

An agent runs forty steps against a real repository. It reports success. Something breaks in production an hour later. Which step did it? For most agent stacks the honest answer is that nobody knows. The trace is a pile of model outputs describing what the agent believed it was doing, the run isn't reproducible, and re-running it produces a different plan anyway. Three properties fix this, and none of them require a better model. Reliability. The agent's world state comes from deterministic sensors, never from the model's own claims. When the model asserts "tests passed" and the sensor disagrees, that's a logged incident, not a state transition. Across an integration suite spanning six system archetypes, 611 of 611 injected false claims were refused, with zero false refusals on honest ones. Reproducibility. No RNG anywhere, ties broken lexicographically, plans replayed bit-exact from a hash-chained journal. Same input, same plan, every time: 120 of 120 generated systems deterministic, 114 of 114 crash-and-resume runs converging to the same state. Auditability. Every step writes an append-only record containing the full decision table that produced it. Tamper with any of it and the chain breaks (3,894 of 3,894 mutations detected). You can ask the system why it chose an action and get the actual reason, not a rationalization generated after the fact. Long horizons are where this pays for itself. At three steps you can eyeball the trace. At forty, determinism is the only thing standing between you and archaeology. I'll show LEDGER running against a real repository, break it on purpose, and replay the failure bit-exact. Then I'll be specific about the cost: what you give up, where the model still genuinely earns its place, and what this approach cannot do.

Bio

Maureese Williams is an AI/Data Engineer at Disney, building RAG and retrieval infrastructure for production systems, with 5+ years across distributed data and ML platforms on AWS and GCP. Co-author of GATS (Graph-Augmented Tree Search, arXiv:2607.08894), a planning framework that replaces per-step LLM calls with search over a persistent, verified world model, with zero model calls at plan time and zero plan variance. Presented the agent-verification side of this work at DEF CON 34.

Sponsors & Partners

Want to become a sponsor? Get in touch!
Let's talk!
We'll email you and share prospectuses for relevant events.
We'd like to (one or more)
Pick at least one
Conferences (one or more)
Pick at least one
Regions (one or more)
Pick at least one
Budget
Pick one