Training-free sparse attention

SparsePR

Partition the Support. Reconstruct the Residual.

Training-Free Sparse Attention for
Video Generation and World Models

Pardis Taghavi · Reza Langari · Gaurav Pandey

Texas A&M University

TL;DRSparsePR couples executable response-aware partitions with a small set of exact probe rows to recover the residual that sparse attention leaves behind.

01 / See the result

SparsePR across four video models.

Cosmos-Predict2.5-14B

Image-to-world · native 720p class

6 VBench + 4 PBench · metric-selected

Prompt

A view of a star trail in the night sky.

VBench40.33 dB

Prompt

An aerial view of a rocky beach in Indonesia.

VBench26.19 dB

Prompt

A butterfly sits on top of a purple flower.

VBench19.82 dB

01 / 10

Cosmos3-Nano-16B

Image-to-world · 720p

6 VBench + 1 PBench sample · metric-selected

Prompt

A highland cow with long horns standing in a field.

VBench26.07 dB

Prompt

A giraffe walking in a field.

VBench22.92 dB

Prompt

A warthog is walking in the grass.

VBench21.41 dB

01 / 07

HunyuanVideo-13B

Text-to-video · 720p

12 quality-selected VBench samples · metric-selected

Prompt

Deep in the dense forest, a lighthouse stands alone, its light at the top flickering on and off. The entire video presents a suspenseful atmosphere.

VBench33.99 dB

Prompt

Two dolphins are swimming in the blue sea.

VBench33.70 dB

Prompt

In the laboratory, Thomas Edison is using a pen to record his experimental data.

VBench32.91 dB

01 / 12

Wan2.2-I2V-A14B

Image-to-video · native aspect ratio

10 selected VBench samples · metric-selected

Prompt

A blue and white smoke is swirly in the dark.

VBench28.73 dB

Prompt

An aerial view of a small town on the edge of the ocean.

VBench27.28 dB

Prompt

Colorful buildings on the seaside cliffs.

VBench28.89 dB

01 / 10

21.9–26.0%executed-pair density

1.48–2.61×end-to-end speedup

4models

0training steps

02 / Abstract

Executable sparsity needs more than concentrated attention.

Training-free block-sparse attention can accelerate video transformers, but attention concentration alone does not define an executable sparse operator. Shared query routes can expand support, and retained attention mass does not predict the output error from skipped interactions.

SparsePR combines Response-Coupled Partitioning with Probe-Fitted Residual Reconstruction. It groups queries and paired K/V tokens by current-call responses, then uses a small set of exact query rows to correct the sparse output. Across four models, SparsePR preserves quality at 22% to 26% executed-pair density with 1.48× to 2.61× end-to-end speedups.

03 / Why partition geometry matters

Why per-query sparsity is not executable sparsity.

Support density is the percentage of key tokens needed to retain 90% of attention mass. The pooled diagnostic measures their union across queries before block routing.

Queries sharing one executable route8

1 query · 6.2%8 queries · 22.9%

Single query

Per-query support

6.2%

One query needs a small key support.

Shared route

8 queries pooled

22.9%

Each query is sparse; a shared route must cover their union.

Route

Omitted key supportRetained key supportPooled support

Per-query sparsity versus pooled support density and normalized output error versus retained attention mass across four models — Per-query concentration does not determine pooled support, and retained attention mass does not determine output error.

04 / Method

One response geometry.
Two coupled stages.

SparsePR constructs the executable partition and reconstructs its error using features from the same current attention call.

01Sample responses

Probe a compact subset of queries against K/V.

→

02Partition support

Form paired K/V groups, then response-aligned query groups.

→

03Execute sparse routes

Select and evaluate hardware-ready query–K/V cells.

→

04Reconstruct residual

Fit a call-specific correction from exact probe rows.

RCP

Response-Coupled Partitioning

Sampled query responses define value-aware paired K/V groups. Their centroids become query-response coordinates, aligning queries that can efficiently share one route.

PFRR

Probe-Fitted Residual Reconstruction

A small stratified set of query rows is evaluated exactly. Their observed post-softmax residuals fit an affine correction in a low-rank probe-residual subspace.

05 / Results

Quality and efficiency across four video models.

All quantitative results from the final draft. Values use matched hardware and sequence shapes where reproduced. Density includes routing and exact probe pairs.

Table 1

Quality and efficiency

Reference fidelity, task quality, executed-pair density, attention PFLOPs, and end-to-end speedup.

Model	Method	PSNR ↑	SSIM ↑	LPIPS ↓	ImgQual ↑	SubCons ↑	PBench ↑	Density ↓	PFLOPs ↓	E2E ↑
HunyuanVideo-13B	Dense	–	–	–	0.850	0.976	–	100%	612.38	1.00×
HunyuanVideo-13B	SpargeAttn†	24.589	0.796	0.232	–	0.908	–	40.09%	389.76	1.38×
HunyuanVideo-13B	SVG2†	30.452	0.910	0.117	0.852	0.927	–	25.45%	299.02	2.30×
HunyuanVideo-13B	SVOO†	24.879	0.843	0.224	0.6793	0.9799	–	–	–	2.17×
HunyuanVideo-13B	SVG-EAR†	31.043	0.928	0.092	0.845	0.903	–	22.17%	281.86	1.93×
HunyuanVideo-13B	SparsePR	31.844	0.932	0.087	0.850	0.976	–	21.92%	255.95	2.61×
Wan2.2-I2V-A14B	Dense	–	–	–	0.689	0.974	–	100%	658.46	1.00×
Wan2.2-I2V-A14B	SpargeAttn†	27.140	0.883	0.116	0.680	0.958	–	30.15%	396.83	1.58×
Wan2.2-I2V-A14B	SVG2†	26.562	0.861	0.138	0.668	0.959	–	31.28%	393.95	1.59×
Wan2.2-I2V-A14B	SVOO†	29.678	0.913	0.095	0.7337	0.9731	–	–	–	1.61×
Wan2.2-I2V-A14B	SVG-EAR†	29.759	0.918	0.093	0.680	0.959	–	23.64%	378.88	1.61×
Wan2.2-I2V-A14B	SparsePR	30.658	0.907	0.044	0.687	0.973	–	21.97%	328.70	1.80×
Cosmos-Predict2.5-14B	Dense	–	–	–	0.714	0.976	77.76	100%	526.87	1.00×
Cosmos-Predict2.5-14B	SVG2	20.075	0.624	0.330	0.678	0.896	76.14	28.81%	286.51	1.24×
Cosmos-Predict2.5-14B	SVOO	22.066	0.685	0.289	0.701	0.909	76.03	37.63%	315.38	1.03×
Cosmos-Predict2.5-14B	SVG-EAR	25.549	0.908	0.062	0.710	0.976	77.78	29.75%	289.69	1.10×
Cosmos-Predict2.5-14B	SparsePR	26.328	0.942	0.068	0.714	0.976	77.75	22.14%	253.61	1.51×
Cosmos3-Nano-16B	Dense	–	–	–	0.700	0.950	77.31	100.00%	90.01	1.00×
Cosmos3-Nano-16B	SVG2	22.458	0.735	0.216	0.677	0.915	75.03	37.29%	57.69	1.16×
Cosmos3-Nano-16B	SVOO	16.642	0.573	0.381	0.707	0.962	77.59	67.32%	69.51	1.02×
Cosmos3-Nano-16B	SVG-EAR	21.167	0.709	0.261	0.658	0.872	72.85	37.18%	57.64	1.10×
Cosmos3-Nano-16B	SparsePR	24.417	0.801	0.176	0.699	0.949	77.30	25.96%	43.22	1.48×

† Reported by prior work. Rows without † are reproduced under matched hardware, sequence shape, and timing protocols.

Error reduction from response-coupled partitioning across four models and Wan2.2 full-generation latency breakdown — Response-coupled partitioning reduces mean and p99 error at 22% density. On Wan2.2, SparsePR reaches 1.80× speedup and probe repair uses 1.1% of total latency.

Table 2

Partitioning and reconstruction ablations

Mean / p99 normalized attention-output error at 22% total executed-pair density. Lower is better.

Configuration	HunyuanVideo	Wan2.2	Cosmos-Predict2.5	Cosmos3-Nano
Semantic partition	0.0887 / 0.7136	0.1634 / 1.7338	0.7903 / 7.5318	0.3590 / 3.3557
Key-response K/V partition	0.0851 / 0.8121	0.1560 / 1.6591	0.7686 / 7.2700	0.3409 / 3.1738
Response-coupled partition	0.0736 / 0.6967	0.1489 / 1.6479	0.7617 / 7.2271	0.3315 / 3.1502
Semantic + probe repair	0.0527 / 0.3562	0.1041 / 0.8186	0.2622 / 0.8260	0.1720 / 0.8648
SparsePR	0.0330 / 0.2285	0.0707 / 0.4305	0.0954 / 0.5769	0.0822 / 0.4951

06 / Citation

Build on SparsePR.

@article{sparsepr2026,
  title   = {Partition the Support, Reconstruct the Residual:
             Training-Free Sparse Attention for Video Generation and World Models},
  author  = {Taghavi, Pardis and Langari, Reza and Pandey, Gaurav},
  journal = {arXiv preprint arXiv:2608.18484},
  eprint  = {2608.18484},
  archivePrefix = {arXiv},
  year    = {2026}
}

SparsePR across four video models.

Cosmos-Predict2.5-14B

Cosmos3-Nano-16B

HunyuanVideo-13B

Wan2.2-I2V-A14B

Executable sparsity needs more than concentrated attention.

Why per-query sparsity is not executable sparsity.

Per-query support

8 queries pooled

One response geometry.Two coupled stages.

Response-Coupled Partitioning

Probe-Fitted Residual Reconstruction

Quality and efficiency across four video models.

Quality and efficiency

Partitioning and reconstruction ablations

Build on SparsePR.

One response geometry.
Two coupled stages.