Ship the Kernels With the Weights:
Compiled GPU Artifacts Are Build Outputs
NVIDIA got model weights into GPU memory in 11 seconds, and startup barely moved. The bottleneck was the kernels every replica compiles for itself. What ModelExpress shows about treating compiled GPU code like any other build artifact.
By Serverless Ventures
Published: October 26, 2026 · Read time: ~9 minutes
The short version
When you distribute an optimized model, distribute its compiled GPU artifacts alongside its weights, and label those artifacts with the hardware and software they are compatible with. Otherwise every consumer pays to compile the same kernels again, or worse, picks up a cache built for a different machine.
NVIDIA's open-source ModelExpress makes the case with numbers: moving DeepSeek-V4-Pro's weights peer-to-peer took startup from 8 minutes to 7. Moving the kernel caches too took it to 1 minute 44 seconds. The lesson generalizes: compiled model artifacts are versioned, environment-specific build outputs, not incidental files on a machine.
The cold start nobody budgets for
Starting an inference replica used to mean one slow thing: getting hundreds of gigabytes of weights from storage into GPU memory. That problem is largely solved. ModelExpress's benchmark loads DeepSeek-V4-Pro on an 8×B200 node like this:
| Weight loading path | Time | vs. cold pull |
|---|---|---|
| Cold pull from Hugging Face | 8m 53s | 1× |
| ModelStreamer from S3 | 3m 16s | 2.7× |
| High-throughput local storage, cold page cache | 1m 10s | 7.6× |
| Peer-to-peer GPU-to-GPU over NIXL/RDMA | 11s | 48× |
Eleven seconds is remarkable. But that table measures how long it takes to load the weights, not how long until the model can serve its first request. During its first forward passes, the engine just-in-time compiles and autotunes kernels with torch.compile, Triton, DeepGEMM, TileLang, CuTe DSL and FlashInfer. Once weights are fast, that compilation is the cold start:
That is a 1.1× speedup from weights alone and 4.6× once kernel caches move too. NVIDIA puts the principle plainly: "When the model, software stack, and GPU architecture match, one replica can pay the compilation cost and the rest can inherit the resulting caches."
Where this matters
Every new replica during a traffic spike pays the compile tax at the worst possible moment. Inheriting caches turns scale-out minutes into seconds.
If you ship an inference stack to customers on different NVIDIA GPUs, precompile for the supported environments instead of making every customer pay the full compilation cost.
Teams or companies running the same model on similar hardware rebuild identical artifacts independently. A shared, versioned artifact removes that waste.
How ModelExpress does it
ModelExpress is a Rust and Python component of NVIDIA Dynamo that sits next to an inference engine. It works with vLLM (natively, as --load-format modelexpress from 0.23.0), SGLang and, in beta, TensorRT-LLM. For compiled artifacts it does four things:
- Finds a ready source. A metadata service (Redis or Kubernetes) lets new replicas discover a replica that already compiled for the same configuration.
- Matches by identity. Each source publishes a source identity (model, revision, engine, parallel layout, dtype, quantization, artifact type) that is hashed into an ID. Only sources with the same ID are candidates.
- Moves the bytes directly. Cache files move host to host over NIXL RDMA in chunks, without a shared filesystem. Each chunk is verified with a CRC32C checksum before it is installed into the engine's own cache directory.
- Falls back safely. If no compatible source exists, the engine compiles as usual, and the new replica can become a source for the next one.
Go deeper: for a full technical breakdown of ModelExpress's architecture, from the Rust server and metadata backends to NIXL transfers, read the Snack on AI deep dive, ModelExpress: NVIDIA Dynamo's Rust-Based Weight Management Layer. Its takeaway matches ours: the JIT cache transfer is the feature nobody is talking about.
From replica-to-replica to a build pipeline
ModelExpress shares artifacts between live replicas inside one cluster. Generalize the idea beyond a cluster, to customers, regions or partner organizations, and it becomes a familiar software pattern: a build and artifact-distribution pipeline, except the outputs include hardware-specific GPU code.
Python solved a version of this years ago with wheels: one package, many prebuilt binaries, each tagged with the interpreter and platform it runs on, and a resolver that picks the right one or builds from source. Compiled model artifacts need the same thing, with tags that describe GPUs and inference stacks instead of operating systems.
The hard part: the cache key
Everything depends on the key that says which artifact fits which machine. Get it wrong in one direction and you recompile for no reason; get it wrong in the other and you load code built for a different environment.
| Dimension | Why it can break a cache | In ModelExpress |
|---|---|---|
| Model and revision | Different weights, different kernels and shapes | In the source identity (model name, pinned revision) |
| Engine and version | vLLM, SGLang and TensorRT-LLM generate different code | Framework and an MX format version are in the identity |
| Parallel layout | Tensor, pipeline and expert parallel sizes change shard shapes | TP, PP and EP sizes are in the identity |
| Precision | bf16 and fp8 compile to different kernels | dtype and quantization are in the identity |
| Accelerator | A cache for one GPU family is useless on another | Checked separately at runtime; mismatched sources are skipped |
| GPU architecture, CUDA, driver, compiler | Generated code targets a specific architecture and toolchain | Not in the identity; the underlying kernel caches carry their own keys |
- Too strict: matching on an exact driver version forces a recompile after every routine driver update, even when the generated code would run unchanged.
- Too loose: matching on GPU architecture alone can reuse artifacts built with a different compiler, engine version or parallel layout.
ModelExpress's design shows the trade-offs. Its metadata documentation notes that the source ID "validates declared identity rather than hashing tensor contents, so the guarantee is only as strong as the revision pin." The GPU family is checked at runtime rather than hashed in, and an empty value is treated as unknown rather than a mismatch, for backward compatibility. Below that, the caches themselves, such as Triton's, carry their own keys that include the target and compiler version, so a mismatch at that level generally means a cache miss and a recompile, not a crash. Layered keys like this are sensible. Just know which layer is protecting you.
When you cannot precompile everything
Not every artifact can be built ahead of time. Some compilation depends on tensor shapes, model configuration or runtime inputs, and autotuning picks kernels by measuring them on the actual hardware. Three patterns cover most cases:
- Precompile the known shapes for the supported environments, and ship them with the release.
- Warm on a seed replica under representative traffic, then publish what it compiled, which is ModelExpress's model.
- Capture on first run and republish, so the next consumer in the same environment starts warm.
In every case the last step must remain the same: if no compatible artifact exists, compile locally and keep serving.
Our read
Once weights move at network speed, inference cold starts look less like data loading and more like compilation. Treat it with build tooling: caches, keys, registries.
Expect model artifacts to ship like wheels: weights plus prebuilt kernels for named GPU and engine combinations, with a fallback to build from source.
These artifacts are executable GPU code. Checksums catch corruption, not tampering; anything shared across organizations should be signed and traceable to its build.
This matters most where replicas come and go: autoscaling fleets, edge sites like the ones in Physical AI Lives Inside the Fence, and small models deployed widely, as in What a 4GB Model Is For. It also cuts the time expensive GPUs sit idle compiling instead of serving, the quiet cost behind Stranded Compute.
Questions for your inference stack
- How long does a new replica take from process start to first request, and how much of that is compilation?
- Which compiled artifacts does your engine produce, and where do they live today?
- What is in your cache key, and what happens when a driver, CUDA or engine version changes?
- Is every artifact tied to a pinned model revision, or to a name that can change underneath it?
- Can a new replica fall back to compiling without failing if the cache is missing or incompatible?
- If you share artifacts beyond one cluster, who signs them, and how would you detect a tampered one?
Sources: ai-dynamo/modelexpress on GitHub (benchmarks, integrations and architecture) and its metadata documentation; NVIDIA Technical Blog, ModelExpress: Distributing Model Artifacts at the Speed of Light, July 24, 2026. Snack on AI, ModelExpress: NVIDIA Dynamo's Rust-Based Weight Management Layer, July 19, 2026. Benchmark figures are NVIDIA's, measured on DeepSeek-V4-Pro with vLLM 0.23.0, TP=8 on an 8×B200 node with ConnectX-7 NICs, and will vary with storage, network and runtime configuration.