Most agent demos are easy to judge right away: either the tool works or it doesn’t, and the answer is either grounded or not. Decision systems are trickier. They mix uncertain forecasts with strict rules, and you only see the results days later as noisy feedback that doesn’t show if the decision was good or just lucky. SmartPlayFPL is an agent-based system for making weekly Fantasy Premier League decisions within a set budget and strict squad rules. Specialist agents review different types of evidence on a shared blackboard. Their recommendations are combined by a weighted consensus engine, which applies rule-based vetoes, and then a constrained optimizer picks the final move. An orchestrator LLM manages the workflow and explains the result, but it does not handle forecasting, enforcing constraints, or optimization. I’ll explain why this separation is important, which failures are due to the model itself, and which happen when the model is given the wrong task. I’ll also discuss how to judge decisions when feedback is delayed, noisy, and only shows what actually happened, not what could have happened. I’ll compare the full system to simpler baselines, like a forecast-plus-optimizer pipeline without an LLM and a single agent with tools, since any extra complexity should be justified. The case study focuses on Fantasy Premier League, but this kind of problem also comes up in portfolio construction, capacity planning, and resource allocation. An LLM can play a key role in a decision system without being the whole system.
Qazybek Beken is a Master of Information Management and Systems candidate at UC Berkeley and the former Head of Data Analytics, Technology and AI at PwC Kazakhstan, where he led machine learning, analytics, and automation work across financial services, retail, and mining. He founded AirData.kz, a public air-quality data platform in Kazakhstan. He is currently building SmartPlayFPL, an agentic decision system combining specialist analysis, deterministic controls, and mathematical optimization.
