CUDA: Fuse MMVQ post-scale for NVFP4 (#24481)

• CUDA: Fuse MMVQ for NVFP4 and BS 1

TODO:

1. Add tests to test-backend-ops (did verify correctness manually for one model)
2. Reorder bias/scale once PRs for NVFP4 are merged/landed

• Add dense MMVQ fusion as well

Perf numbers on B4500. Note qwen35 is FP8->Q8

• ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite

  Model	Test	t/s master	t/s osimons/nvfp4_fuse_mmvq	Speedup
  qwen35moe 35B.A3B NVFP4	tg128@d32768	150.15	156.29	1.04
  qwen35moe 35B.A3B Q4_K_M	tg128@d32768	157.91	157.64	1.00

Perf numbers on DGX Spark

• ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite

  Model	Test	t/s master	t/s osimons/nvfp4_fuse_mmvq	Speedup
  qwen35moe 35B.A3B NVFP4	tg128@d32768	58.31	59.69	1.02
  qwen35moe 35B.A3B Q4_K_M	tg128@d32768	54.94	54.79	1.00

• Add tests for the added fusion ops

• Cleanup test-backend-ops

• Cleanup ggml-cuda/mmvq

1. Unrestrict post-scale fusion
2. Rename names accordingly
3. Remove env variable to disable fusion

• Merge old mul_mat patterns into the lane-based approach

• Enable fusion for MoE in shared MMVQ

• Restrict scale_view_nodes, enroll MM + ADD into lane-matcher

• Refactor mmvq loads, still does not help non-nvfp4 kernels

• Restrict scale-fusion to NVFP4

This is necessary, as the prolog is quite heavy in GEMV for some quants/model configs, leading to net perf regression. We should really be looking to refactor this such that ratio of prologue/hot-loop/epilogue is better on the hot-loop front:

• ./scripts/compare-llama-bench.py -b master -c c1b9381d327e063cc846b46b59708444b66dc4d8 --tool llama-bench -i llama-bench.sqlite

  CPU	Model	Test	t/s master	t/s c1b9381d3	Speedup
  INTEL(R) XEON(R) GOLD 6542Y	gemma4 26B.A4B NVFP4	tg128@d32768	151.70	154.32	1.02
  INTEL(R) XEON(R) GOLD 6542Y	gemma4 26B.A4B Q4_K_M	tg128@d32768	187.95	185.73	0.99
  INTEL(R) XEON(R) GOLD 6542Y	gpt-oss 20B MXFP4 MoE	tg128@d32768	304.62	300.69	0.99
  INTEL(R) XEON(R) GOLD 6542Y	qwen35moe 35B.A3B NVFP4	tg128@d32768	193.72	211.99	1.09
  INTEL(R) XEON(R) GOLD 6542Y	qwen35moe 35B.A3B Q4_K_M	tg128@d32768	217.76	218.15	1.00

• Reorder scale & bias-add to adhere to #24331

• Restrict lane scale to NVFP4

Don't need to test unfused combinations

• Cleanup

• Merge single-lane mm-fusion helpers

• Refactor and clean-up host-side fusion logic

• Move gate_bias and scale into the same active-thread guard

Latest perf numbers: B6000

build: 5b7d9f272 (9578)

• ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite

  CPU	Model	Test	t/s master	t/s osimons/nvfp4_fuse_mmvq	Speedup
  INTEL(R) XEON(R) GOLD 6542Y	gemma4 26B.A4B NVFP4	tg128@d32768	151.79	154.10	1.02
  INTEL(R) XEON(R) GOLD 6542Y	gemma4 26B.A4B Q4_K_M	tg128@d32768	187.90	187.27	1.00
  INTEL(R) XEON(R) GOLD 6542Y	gpt-oss 20B MXFP4 MoE	tg128@d32768	303.77	306.56	1.01
  INTEL(R) XEON(R) GOLD 6542Y	qwen35moe 35B.A3B NVFP4	tg128@d32768	193.41	207.99	1.08
  INTEL(R) XEON(R) GOLD 6542Y	qwen35moe 35B.A3B Q4_K_M	tg128@d32768	217.60	218.58	1.00

DGX Spark

build: 5b7d9f272 (9578)

• ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite

  CPU	Model	Test	t/s master	t/s osimons/nvfp4_fuse_mmvq	Speedup
  CPU	gemma4 26B.A4B NVFP4	tg128@d32768	34.61	34.84	1.01
  CPU	gemma4 26B.A4B Q4_K_M	tg128@d32768	46.95	46.90	1.00
  CPU	gpt-oss 20B MXFP4 MoE	tg128@d32768	64.84	64.62	1.00
  CPU	qwen35moe 35B.A3B NVFP4	tg128@d32768	59.63	60.72	1.02
  CPU	qwen35moe 35B.A3B Q4_K_M	tg128@d32768	56.53	56.55	1.00

PPL values for 5 chunks: this PR

model mode ppl uncertainty log /mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf fusion_enabled 5.2892 0.35389 ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_enabled.log /mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf fusion_disabled 5.2742 0.35215 ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_disabled.log /mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf fusion_enabled 5.4487 0.36866 ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_enabled.log /mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf fusion_disabled 5.4403 0.36782 ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_disabled.log /mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf fusion_enabled 17342.4348 3703.13932 ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_enabled.log /mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf fusion_disabled 18627.0624 3998.42475 ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_disabled.log /mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf fusion_enabled 363.8913 33.14007 ppl-value-checks/gpt-oss-20b-mxfp4.fusion_enabled.log /mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf fusion_disabled 363.8913 33.14007 ppl-value-checks/gpt-oss-20b-mxfp4.fusion_disabled.log /mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf fusion_enabled 17330.3926 3716.70472 ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_enabled.log /mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf fusion_disabled 17933.9524 3883.17066 ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_disabled.log

master: summary: ppl-value-checks/summary.tsv model mode ppl uncertainty log /mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf fusion_enabled 5.2892 0.35389 ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_enabled.log /mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf fusion_disabled 5.2742 0.35215 ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_disabled.log /mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf fusion_enabled 5.4487 0.36866 ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_enabled.log /mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf fusion_disabled 5.4403 0.36782 ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_disabled.log /mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf fusion_enabled 17342.4348 3703.13932 ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_enabled.log /mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf fusion_disabled 18627.0624 3998.42475 ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_disabled.log /mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf fusion_enabled 363.8913 33.14007 ppl-value-checks/gpt-oss-20b-mxfp4.fusion_enabled.log /mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf fusion_disabled 363.8913 33.14007 ppl-value-checks/gpt-oss-20b-mxfp4.fusion_disabled.log /mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf fusion_enabled 17330.3926 3716.70472 ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_enabled.log /mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf fusion_disabled 17933.9524 3883.17066 ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_disabled.log

• Allow views to weights in ggml_can_fuse_subgraph

• Remove gate_first from test_mul_mat_vec_fusion

• Ditch lane-parsing approach in favor of hard-coded patterns

• Apply suggestions from code review

Co-authored-by: Georgi Gerganov [email protected]

• Rename ggml_is_constant_view_src to ggml_is_constant

• Finish renaming of 0905129e9d12e2bc6f16d6d3cc4e6b40606fc893

• Readd descriptive prints for fusion debugging

• Add weight-buffer pre-allocation to test_case

This is required so we correctly test fusion of NVFP4.

• Update ggml/src/ggml.c

Co-authored-by: Johannes Gäßler [email protected]

• Add 2nd context for weights as suggested by @JohannesGaessler

This reflects more natural use of ggml compared to artifically pre-allocating weights into the same context

• Exclude fused tests from gradient mode

I'm unsure of the current state, but naively every fusion pattern should require its own backpropagation implementation. I don't see these implemented for the CUDA backend, so we can disable tests to avoid triggering GGML_ASSERT for

ggml_tensor * build_graph(ggml_context * ctx) override {
    GGML_ASSERT(!use_weight_context());
    return build_graph(ctx, nullptr);
}

• Apply suggestions from code review

Co-authored-by: Johannes Gäßler [email protected]

Co-authored-by: Georgi Gerganov [email protected] Co-authored-by: Johannes Gäßler [email protected]