Release Notes
opencl: Q6_K GEMM/GEMV fix for ne01 of weights that are not multiples of 128. (#25464)
- opencl: fix garbled output for Q6_K weights with ne01 % 128 != 0 on Adreno
Observed with granite-3.1-3b-a800m-instruct, whose vocab is an odd number.
Route Q6_K dense mul_mat with ne01 % 128 != 0 off the noshuffle path: decode (ne1==1) uses the correct flat GEMV and the matching GEMM (ne1>1) falls back to CPU (the flat convert has no verified small-batch GEMM kernel for these shapes). All standard hidden/FFN/vocab dims are multiples of 128 and keep the noshuffle path.
- opencl: reserve alignment slack for the SOA subbuffer carve in alloc size
set_tensor carves quantized weights into per-component subbuffers (d/q, ql/qh/s/d, ...) whose origins are each rounded up to the device base address alignment. When a component's size is not a multiple of the alignment, the carve extends past ggml_nbytes(tensor) and the last subbuffer overlaps the next tensor in the pool -- e.g. q6_K [1536, 49155]: size_s = 49155*96 ends 32 bytes past a 128-byte boundary, so the d subbuffer ends 96 bytes past the tensor's allocation, and whichever of the two neighboring tensors is uploaded last silently corrupts the other (here: the last vocab rows' block scales). This affects any quant type whose component sizes can be misaligned, on any shape with ne01 not a multiple of the alignment granularity; standard power-of-two dims are unaffected.
Implement get_alloc_size for the OpenCL buffer type and reserve the worst-case carve slack (4 aligned gaps; 5 components max, q5_K) for quantized tensors. Costs at most 512 bytes per quantized tensor at the observed 128-byte alignment.
- opencl: use lm based q6_k mm when ne1 is not multiple of 128
Co-authored-by: Li He [email protected]
macOS/iOS:
- macOS Apple Silicon (arm64)
- macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
- macOS Intel (x64)
- iOS XCFramework
Linux:
- Ubuntu x64 (CPU)
- Ubuntu arm64 (CPU)
- Ubuntu s390x (CPU)
- Ubuntu x64 (Vulkan)
- Ubuntu arm64 (Vulkan)
- Ubuntu x64 (ROCm 7.2)
- Ubuntu x64 (OpenVINO)
- Ubuntu x64 (SYCL FP32)
- Ubuntu x64 (SYCL FP16)
Android:
Windows:
- Windows x64 (CPU)
- Windows arm64 (CPU)
- Windows arm64 (OpenCL Adreno)
- Windows x64 (CUDA 12) - CUDA 12.4 DLLs
- Windows x64 (CUDA 13) - CUDA 13.3 DLLs
- Windows x64 (Vulkan)
- Windows x64 (OpenVINO)
- Windows x64 (SYCL)
- Windows x64 (HIP)
openEuler:
- DISABLED
- openEuler x86 (310p)
- openEuler x86 (910b, ACL Graph)
- openEuler aarch64 (310p)
- openEuler aarch64 (910b, ACL Graph)
UI: