b10076
b10076
View on GitHubView PackagePublished: Jul 21, 2026

Release Notes

CUDA: vectorize same-type get_rows with int4 copy (#25929)

k_get_rows_float did a scalar one-element-per-thread copy and recomputed the row-invariant work (index load, fast_div_modulo, src/dst row pointers) for every element. Hoist that out of the per-element loop, and add a vectorized path (k_get_rows_float_vec) that copies one int4 (16 B) per thread for the contiguous same-type (no-cast) case.

The vectorized path is gated at compile time (is_same<src0_t, dst_t>) and at runtime on 16-byte alignment of the base pointers and all row strides and on ne00 % VEC == 0. Vectorizing divides the block count by VEC, so a small single-row gather can drop below the device CU count and regress; an occupancy gate keeps those on the block-rich scalar path.

On Strix Halo (gfx1151) the DeltaNet recurrent-state gather (ne00=524288) drops 18.6us -> 13.0us (rocprofv3 HW timestamps), faster than the Vulkan backend, with no regression on the small conv-state gather; total get_rows -27%. test-backend-ops GET_ROWS passes (47/47).

Assisted-by: Claude Opus 4.8

Website:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI: