vLLM 0.32 adds smaller KV caches, faster start-up and security fixes
What changed
vLLM, the open-source engine many teams use to serve language models, released version 0.32.0, with work from 363 contributors. The notes highlight KV cache formats that use less memory on more kinds of GPU, new start-up commands, support for new models, and fixes for three security advisories. A browser-origin default was also tightened.
Why it matters for builders
The KV cache is the memory that grows with every user and every token, and it often limits how many people one GPU can serve. A smaller one can mean more users per GPU. Faster start-up helps scaling, because a slow new replica cannot take traffic. If you expose the server, read the security items.
What to check
The speed-ups quoted are from the project's own tests on specific models and chips. One setting trades a small rise in time to first token for more speed. Test on your own model and hardware, and read the release notes before you upgrade.