AI Models Can Signal Through Shared Inference Servers
Researchers showed that models sharing a vLLM server can pass bits through batching and KV cache effects, even when sandboxed from the network.
Researchers showed that models sharing a vLLM server can pass bits through batching and KV cache effects, even when sandboxed from the network.
Coding agents wrote one GPU program per model that runs the whole decode loop. On eight open models it beat the fastest general engine by up to 2.05x.
Startup Standard Kernel says it tuned Intel’s Gaudi2 to beat Nvidia’s H100 on long-context inference, with 60% higher decode throughput at a fraction of the rental price.
Researchers decoded a SASS control code field used in cuBLAS and cuDNN but never emitted by nvcc, and gained up to 31.9% on kernels.