If you’re being challenged by deploying artifacts for AI workloads on a platform built for lightweight, ephemeral containers, you’re not alone. Model deployments are coming with 20-minute cold start times, nodes OOMing, and downloads restarting due to pod failures. We’ll unpack all the solutions out there, starting with level 1 - implementing an in memory caching solution. As we progress the levels get more intricate, such as leveraging kServe cluster storage containers and HuggingFace XET proxies. The final level includes ModelPack, Harbor, and Spegel - which can either supercharge or super-nerf your approach. To finish the game we will answer the most important questions of them all: “Can model artifacts be converted into OCI artifacts, and if so, at what cost?” You’ll leave knowing which solution can be implemented into your platform today, and the cheat codes to complete the next levels of solutions. It’s dangerous to deploy model artifacts alone, take our lessons with you.
Philip Laine is the founder of Kvick a company making pod-startup times a problem of the past. He runs Spegel, a stateless cluster-local peer-to-peer OCI registry mirror for Kubernetes, and is a former core maintainer of Flux and Zarf.
