A new pixel diffusion decoder called PiD turns the compact latents of a text-to-image model straight into high-resolution pictures. It replaces the usual two-step routine of decoding a small image and then running a separate super-resolution model. On a consumer RTX 5090, PiD decodes a 512×512 image’s latent into a 2048×2048 picture in under one second.
The researchers report that it is about six times faster than cascaded diffusion-based super-resolution pipelines, with better visual fidelity.
Why decoding became the bottleneck
Most practical high-resolution text-to-image systems use latent diffusion. The model generates in a compressed latent space, where the math is cheap, and a decoder then maps that latent back into pixels.
The PiD authors point to two weaknesses in that last step. The standard decoder is built to reverse the encoder, so it reconstructs what the latent holds and doesn’t invent extra detail. And it gets expensive at megapixel sizes. To reach 4K-class output, many pipelines bolt on a separate upscaling model, which adds both time and another place for errors to creep in.
How PiD works
PiD reframes decoding as conditional pixel diffusion. Instead of reconstructing, it denoises directly in high-resolution pixel space while being guided by the latent. Decoding and upsampling become a single generative module, and the system can produce 4x and even 8x upscaled images with low latency.
A lightweight adapter, which the authors call sigma-aware, feeds noise-corrupted latents into the pixel diffusion backbone. That has a practical side effect: PiD can decode latents that are only partly denoised. The latent diffusion process can therefore stop early, saving generation steps before decoding begins.
To cut cost further, the team distilled the model with DMD2, which brings inference down to four steps.
Works with different latent types
PiD isn’t tied to one kind of latent. It handles conventional VAE latents and the semantic latents used in recent RAE-based models, such as those built on SigLIP or DINOv2 features. That matters because the field is moving toward different encoders, and a decoder that only works with one of them would age quickly.
The numbers
The speed figures come from two different kinds of hardware, a consumer card and a data-center chip.
- 512×512 latents decode to 2048×2048 pixels in under 1 second on an RTX 5090, using 13 GB of peak memory
- As fast as 210 ms on a GB200 GPU
- About 6x faster than cascaded diffusion-based super-resolution pipelines
- Up to 5.9x faster than SeedVR2 in the decoding-plus-upsampling comparison, at 211.2 ms against 1237.5 ms
The project page also reports how often human judges preferred PiD’s output over each baseline, alongside side-by-side 4K decoding examples. The authors say the visual fidelity is better than the cascaded approach, though those preference tests are the team’s own and haven’t been independently repeated.
What this means for image generation tools
Thirteen gigabytes of peak memory is within reach of a high-end consumer graphics card, which suggests the approach could run on a desktop as well as in a data center. For anyone building an image generator, a faster final stage also gives room to spend the saved time elsewhere, for example on more sampling steps in the base model or on serving more users per GPU.
Because the decoder accepts partly denoised latents, it also offers a way to shorten the whole pipeline and not only the last stage. Stopping the latent process early and letting PiD finish the job could trim latency on every image.
What to look for next
The results come from the authors’ project page, so the next test is how PiD behaves when other teams plug it into their own models. Useful checks include whether quality holds on unusual styles and text-heavy images, and whether the four-step distilled version keeps its detail at the 8x setting.
If the speed and quality claims hold up outside the lab, a one-stage pixel diffusion decoder could become the default last step for high-resolution latent diffusion systems, replacing the decode-then-upscale chain many of them use today.
Source: research.nvidia.com