Superpower Daily
Judge Vacates Pentagon’s Anthropic Blacklist and Orders Defense...Databricks Files $5B Equity Offering With Nearly All of It Alre...Snowflake Open-Sources Semi-Persistence With Sub-Second Single-...Cisco Rolls Out MyAgent to 90,000 Employees for Supervised Cros...Midjourney Opens V8.2 Image-Editing Tests With Up to Four Refer...Gemini Omni 1.1 Flash Adds 40-Second Scene Extensions and 4K Fi...Judge Vacates Pentagon’s Anthropic Blacklist and Orders Defense...Databricks Files $5B Equity Offering With Nearly All of It Alre...Snowflake Open-Sources Semi-Persistence With Sub-Second Single-...Cisco Rolls Out MyAgent to 90,000 Employees for Supervised Cros...Midjourney Opens V8.2 Image-Editing Tests With Up to Four Refer...Gemini Omni 1.1 Flash Adds 40-Second Scene Extensions and 4K Fi...
Toolspublished

Snowflake Open-Sources Semi-Persistence With Sub-Second Single-GPU vLLM Swaps

The release treats GPU model copies as disposable while retaining weights in CPU memory, aiming to let many specialized models share costly accelerator capacity without long reload delays.

By 3 min read
Snowflake Open-Sources Semi-Persistence With Sub-Second Single-GPU vLLM Swaps
Sign in to save and react

Listen to this story

The audio brief

About 1:34
0:001:34
Read transcript
Snowflake has open-sourced a way to swap AI models on and off GPUs in as little as 214 milliseconds, without reloading their weights from storage. The software, called Semi-Persistence, is designed for vLLM-based serving systems that need to run many specialized models on limited accelerator capacity. The basic idea is to treat the copy on the GPU as disposable, while keeping the model’s weights in pinned CPU memory. When demand changes, Semi-Persistence restores those weights in parallel across the server. That avoids both of vLLM’s usual tradeoffs: copying weights off the GPU when a model sleeps, or freeing memory immediately and paying the longer storage reload when it wakes. In Snowflake’s internal tests, single-GPU sleep-and-wake cycles took 214 to 801 milliseconds, compared with 1.2 to 13.5 seconds for vLLM Level 2. Multi-GPU models took 1.75 to 7 seconds, versus 10.5 to 40.6 seconds. Across models ranging from 2 billion to 397 billion parameters, Snowflake reports a 5.6- to 19.9-fold latency reduction. It also reports restoring trillion-parameter models in 32.8 seconds, against cold starts of roughly 13 to 15 and a half minutes. The constraint is that these are internal results from one AWS server configuration with eight H200 GPUs, while the orchestration layer is still experimental. The key question is how those gains hold up across real production workloads and hardware.

Story brief

3 key points

Snowflake’s newly open-sourced Semi-Persistence adds a vLLM-based way to park GPU model instances without copying weights back to GPU or reloading them from storage. It retains weights in pinned CPU memory and restores them in parallel across a node, aimed at serving multiple models under changing demand. In Snowflake’s internal tests, single-GPU cycles took 214–801 milliseconds versus 1.2–13.5 seconds for vLLM...

  1. 01

    Semi-Persistence caches model weights in vLLM’s native format while discarding GPU copies during idle periods.

  2. 02

    For multi-GPU models, Snowflake reports 1.75–7 seconds versus 10.5–40.6 seconds with vLLM Level 2.

  3. 03

    Tests across 2B–397B models showed a 5.6×–19.9× reduction in end-to-end sleep-and-wake latency.

Idle models can give back GPU memory only if they can return before the next request waits too long. Snowflake says its newly open-sourced Semi-Persistence cut single-GPU sleep-and-wake cycles to 214–801 milliseconds in internal tests, making faster model swapping available to vLLM-based serving stacks.

Semi-Persistence keeps a long-lived copy of model weights in a pinned CPU-memory pool while GPU copies are discarded and restored as demand changes. Snowflake released the software through ArcticInference/semi_persistence, alongside an experimental orchestrator and dashboard for scheduling many model instances.

Avoiding the slow path on every swap

The target is a tradeoff in vLLM, the inference engine Semi-Persistence currently wraps. Its Level 1 sleep mode copies weights from GPU to CPU before freeing the GPU, which makes waking faster but adds a full transfer when the model sleeps. Level 2 releases GPU memory immediately, then reloads weights from storage before service resumes.

Snowflake instead separates a reusable model skeleton, meaning its structure and runtime state, from the weights that define a specialization. It caches the skeleton once and retains weights in vLLM’s native format, so a model can be restored without copying weights back from a GPU or loading them again from storage.

Using the whole node to restore weights

The restore path shards weights across the node, transfers those shards in parallel over PCIe, and uses NVLink to assemble them on target GPUs. The system also supports asynchronous loading, eviction, migration, consolidation, and request pausing, so data movement can overlap with request processing.

Its reservation system divides GPU memory into logical slots ranging from fractions of a GPU to several GPUs. An instance without an appropriate slot waits; when one is freed, it is assigned to a waiting instance. That coordination lets instances make independent moves without overcommitting GPU memory.

Fast results on one tested configuration

Across 2B- to 397B-parameter models, Snowflake reports a 5.6×–19.9× reduction in end-to-end sleep-and-wake latency against vLLM Level 2 under otherwise identical configuration. The internal benchmark used an AWS p5en.48xlarge with 192 vCPUs, 2 TiB of memory, and eight H200 GPUs.

Snowflake also reports restoring trillion-parameter models in 32.8 seconds, compared with vLLM cold starts of 13.3–15.5 minutes. The figures are Snowflake-reported results from its tested system.

Sources

  1. snowflake.comSemi-Persistence: Fast Model Swapping for vLLM & GPUs

Related coverage

Continue reading