Self-managed AI model deployments on-prem and in private cloud infrastructure are becoming increasingly popular, accelerated by open-weight models that rival frontier models. But deploying them needs accelerators (such as GPUs) with large and expensive memory. The usual ways to save that memory quantize or approximate the model, thereby changing its output and affecting accuracy. This talk removes that tradeoff. It covers lossless compression of both model weights and the KV cache, across BF16, FP16, FP8, and FP4, delivering around 30% more GPU headroom with identical model output. Weights are bit-for-bit exact with cryptographic hash verification, and stay compressed in GPU memory while they run. Freeing weight memory leaves more room for the KV cache, and the KV cache is then compressed on top. In deployments that are already memory-constrained, this results in cases of up to 110% more KV cache headroom, raising usable context length, batch size, and concurrency. The talk frames bit-exact KV cache compression as the next lever for lossless inference efficiency.
Kunle Olutomilayo is the Founder of Isiro AI, lowering the cost of ownership for self-managed AI infrastructure. The company focuses on lossless reduction of the hardware memory required for model deployments, without changing the model output. Kunle's core thesis is that the next wave of inference cost savings will not only come from faster chips or smaller models, but also from kernel software breakthroughs that move less data to address the memory wall problem, without changing what the model produces. This is especially important in regulated and high-stakes environments, such as finance, health care, defense, and enterprise AI, where cost savings must not come at the expense of predictable model behavior. Kunle holds a PhD in Electrical Engineering and previously worked on advanced AI and vehicle autonomy at Ford Greenfield Labs.
