A straightforward CUDA matrix multiplication is a good correctness baseline; making it fast means changing how data is moved and reused, how work is divided across threads, and how the kernel fits the GPU. For matrices A (M×K) and B (K×N), the result C is M×N. Each C element is the dot product of one row of A and one column of B.
This is a learning progression, not a report of personal code or measurements: no author-specific GPU, implementation, or benchmark results are established here. NVIDIA’s examples illustrate the engineering choices and their workload-specific effects.
Start with the direct mapping: one output per thread
The mathematical definition translates naturally into a kernel: assign each thread an output coordinate (row, column), accumulate the K products for that coordinate, and write the result to C. In pseudocode, each thread computes C[row, col] = sum(A[row, k] * B[k, col]) for k from 0 to K−1.
This baseline makes the indexing and correctness conditions visible. Check that the output has M rows and N columns, that each input access respects its row stride, and that dimensions that are not multiples of a future tile size are handled. Compare results against a trusted implementation, with tolerances appropriate to the accumulation type and input data type.
#1 Best Overall
The drawback is that neighboring threads and successive iterations may fetch the same input values independently. The arithmetic is simple; the challenge is arranging memory access and reuse so the GPU does not spend disproportionate time moving data.
Why memory access changes the result
Coalesce global-memory requests
Coalescing describes how memory requests made by threads in a warp are combined into transactions. A mapping in which adjacent threads access adjacent addresses can use memory transactions more efficiently than one in which threads jump across memory. The best mapping depends on which matrix dimension is contiguous in memory and on whether threads are assigned across output rows or columns.
NVIDIA’s CUDA C++ Best Practices Guide 13.4 gives an illustrative C=AB progression on Tesla V100: its unoptimized example reports 119.9 GB/s effective bandwidth; staging a tile of A in shared memory reports 144.4 GB/s; and also using shared memory to avoid redundant transfers of a tile of B reports 195.5 GB/s. These are effective-bandwidth results for the guide’s code and V100 example, not promised speedups for other GPUs or matrix shapes. The guide’s shared-memory discussion explains the optimization context.
Rank #2
Stage and reuse data in shared memory
Threads in a block can cooperatively load tiles of A and B from global memory into shared memory, synchronize, compute partial products from those tiles, and repeat across K. Reuse is the point: values brought into shared memory can contribute to multiple output elements before the next global-memory load. A tiled kernel therefore trades a simple direct mapping for explicit loading, coordination, and accumulation.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Shared memory is not automatically beneficial. The tile must fit the available resources; accesses must avoid bank conflicts; and synchronization must ensure data is ready before consumption. In some patterns, a shared-memory layout also lets the kernel load global data coalescently and then rearrange it for efficient computation.
Transpose-like access patterns reveal layout costs
Matrix multiplication variants can behave very differently when memory layout changes. NVIDIA’s same guide reports 12.8 GB/s effective bandwidth for its unoptimized C=AAᵀ example on Tesla V100, 140.2 GB/s after using shared memory for coalesced reads, and 199.4 GB/s after removing shared-memory bank conflicts. These numbers belong to that separate transpose-oriented example and must not be treated as directly comparable to the C=AB measurements above. They show why both global access patterns and shared-memory layout matter.
Choose tiles for the whole GPU, not just fewer loads
Tiling is hierarchical. A threadblock works on an output region, warps divide that region into smaller work units, and individual threads hold accumulators or fragments in registers. NVIDIA’s CUTLASS documentation describes this decomposition and its relationship to shared-memory staging, register use, and parallel execution. As CUTLASS puts it, “The basic triple loop nest computing matrix multiply may be blocked and tiled to match concurrency in hardware, memory locality, and parallel programming models.” See the CUTLASS Efficient GEMM documentation.
Larger tiles can increase reuse and reduce global-memory fetches, but they can also consume more shared memory and registers. If M or N is small, a large output tile may waste threads or produce too few threadblocks to occupy the GPU. Smaller tiles can expose more parallel work, but may repeat loads or increase overhead. There is no universal tile size: shape, data type, architecture, and resource use all affect the choice.
- Output shape: Consider whether M and N provide enough independent tiles to launch across the device.
- K dimension: The reduction depth determines how many tile iterations and synchronization points are needed.
- Registers and occupancy: More per-thread accumulators can reduce repeated work, but register pressure can limit the number of active warps.
- Shared memory: Staging helps reuse only when the working set fits and the access pattern avoids bank conflicts.
- Parallelism: A kernel needs enough independent blocks or other scheduled work to keep the GPU busy.
Correctness includes boundaries, synchronization, and precision
Real dimensions often do not divide evenly by tile dimensions. A tiled kernel must mask or otherwise safely handle loads and stores outside M, N, or K, so edge tiles neither access invalid memory nor omit valid outputs. All threads that participate in shared-memory staging must reach the relevant synchronization points consistently; consuming a tile before its loads are complete creates a race.
Accumulation precision also affects the answer. A faster instruction path may use a different multiplication or accumulation format than a simple baseline. Validate representative shapes and values against a trusted reference, and set tolerances that reflect the intended data type and numerical behavior rather than assuming bit-for-bit equality.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Measure each change under stated conditions
A performance result is meaningful only alongside its context. Record the GPU, driver and CUDA toolkit versions, matrix dimensions, input and accumulation data types, baseline implementation, warmup and timing method, and whether transfers are included. Compare like with like: effective bandwidth, elapsed time, and relative throughput are different metrics, and results from different examples do not establish a universal ranking.
For each change, first confirm correctness, then measure the same workload. Vary tile shape and work per thread deliberately; inspect whether the kernel is limited by memory traffic, synchronization, occupancy, insufficient parallel work, or arithmetic throughput. A change that helps a large square matrix may hurt a narrow or irregular one.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteWhen to use a library or a higher-level kernel model
CUTLASS
CUTLASS provides reusable building blocks and GEMM abstractions for high-performance matrix multiplication across NVIDIA architectures and data types. Its September 2026 overview identifies version 4.8.0 and support spanning Volta through Blackwell. That breadth does not mean every architecture-specific kernel is interchangeable: the overview distinguishes Blackwell data-center SM100 from GeForce RTX 50-series SM120 targets. Check the library’s current compatibility guidance for the GPU and toolkit you intend to use. See the CUTLASS overview.
cuTile
NVIDIA’s cuTile tutorial demonstrates assigning output tiles to blocks, iterating through K, using a matrix multiply-accumulate operation, and storing the result. The tutorial states requirements of CUDA 13.1 or later, Blackwell hardware, and Python 3.10 or later; its described optimization support is limited to Blackwell compute capabilities 10.x and 12.x. These are the tutorial’s stated requirements, so check current release documentation before setting up a project. The tutorial reports that its cuTile implementation reaches more than 90% of PyTorch calling cuBLAS performance at large matrix scales on a GeForce RTX 5080; this is a tutorial-reported comparison for its implementation and benchmark conditions, not a general performance guarantee. Read NVIDIA’s CUDA Tile matrix multiplication tutorial.
A custom kernel is useful when learning the mapping or meeting a workload-specific need. For production, compare it against a maintained library on the actual shape and hardware: libraries encode architecture-specific choices, while a hand-written kernel carries the burden of edge cases, resource tuning, and compatibility.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
