Super Micro Computer launched a new KV (key-value) cache storage solution, developed in partnership with IBM and NVIDIA. This collaboration accelerates AI inference performance, critical for demanding large language model (LLM) workloads.
The solution integrates Supermicro's storage servers with IBM and NVIDIA software. It moves reusable AI inference context to a shared storage tier beyond the GPU's memory. This method boosts efficiency and significantly reduces response times. Tests show up to a 56 times faster "Time-to-First-Token" for long-context prompts.