Large Language Models routinely suffer from calibration issues, confidently asserting falsehoods when faced with out-of-distribution tasks. Traditional calibration methods often rely on heavy post-hoc tuning or fragile prompt engineering. In this talk, we explore a more dynamic approach: Self-Calibrating Language Models via Test-Time Discriminative Distillation.We will dive into how we can leverage the "generation-discrimination gap" to train models to double-check themselves on the fly. By utilizing lightweight parameter updates (via LoRA) on late transformer layers during the inference phase, we can align a model's verbalized confidence with its underlying discriminative signals ($P(\text{True})$). Finally, we will discuss the practical engineering trade-offs of this approach, including how to implement entropy-based gating mechanisms to manage computational overhead, making self-calibration viable for real-world deployment.
Rissal Hedna is an AI Engineer at Pharos Labs and a Language AI Researcher specializing in Large Language Model (LLM) calibration, alignment, and parameter-efficient fine-tuning. Working on model reliability, and co-creator of SECL, a novel test-time training framework designed to stop LLMs from confidently lying by forcing them to self-calibrate during inference. Attempting to find direct links between deep machine learning theory and production-ready engineering, he focuses on building AI systems that are mathematically sound, computationally efficient, and safe for critical-domain deployment.
