https://github.com/triton-lang/triton/pull/10321 fixes the second-half alignment of unpacked 16-row TMEM scale stores. The fix is present in XLA's Triton pin, but tmemfix.patch reverts it because XLA uses CUDA 12.8, where ptxas fails with a zero second-half offset. This patch applies an alternative fix that works with CUDA 12.8. It can be removed once the upstream Triton fix is enabled in XLA. --- a/lib/Dialect/TritonNvidiaGPU/IR/Dialect.cpp +++ b/lib/Dialect/TritonNvidiaGPU/IR/Dialect.cpp @@ -101,6 +101,10 @@ if (ll.getBasis(kRow, llvm::Log2_32(16)) == ArrayRef{0, 0}) { nRow /= 2; } + // Scale stores may need a padded half when lowered to 16-row TMEM messages. + // Unpacked store further doubles the padding requirement. + if (isa(memDescType.getEncoding())) + nCol = std::max(nCol, 4); // If multibuffering is present, we need to allocate more cols if (memDescType.getRank() > 2) { assert(memDescType.getRank() == 3); --- a/lib/Dialect/TritonNvidiaGPU/IR/TensorMemoryUtils.cpp +++ b/lib/Dialect/TritonNvidiaGPU/IR/TensorMemoryUtils.cpp @@ -237,8 +237,10 @@ secondHalfOffset = (row << 16) | col; if (*secondHalfOffset == 0) { // Workaround for ptxas bug, we cannot use secondHalfOffset = 0 to write - // only 16 elements. We use secondHalfOffset = 1 instead and we pad the - // allocation. + // only 16 elements. For unpacked scale stores, each source register + // spans two TMEM columns, so place the padded half at the next + // two-column group. Scale TMEM allocations reserve at least four + // columns for this padding. if (!isScales) { if (emitError) { emitError() @@ -246,7 +248,7 @@ } return failure(); } - secondHalfOffset = 1; + secondHalfOffset = unpacked ? 2 : 1; } // We "quotient it out", meaning we remove the last basis from reps auto basis = reps.getBases();