To call FlashAttention-2 from PyTorch, install the FlashAttention package and call its Python functions, such as flash_attn_func, on GPU tensors. A Triton implementation is a different artifact. The Triton fused-attention tutorial is kernel code you read, run and modify to learn how the algorithm maps onto GPU work. Using the package from PyTorch does not require writing or running Triton code yourself, and no single kernel is fastest on every GPU and workload.
Three layers with three different jobs
Much of the confusion around this topic comes from treating “FlashAttention-2”, “the FlashAttention repository” and “the Triton tutorial” as interchangeable. They are not.
As an Amazon Associate I earn from qualifying purchases.
| Layer | What it is | Where to read it |
|---|---|---|
| Algorithm | Exact attention, designed around GPU memory traffic and how parallel work is partitioned | FlashAttention-2 paper (arXiv 2307.08691) |
| Triton implementation | The fused-attention sample in the Triton documentation, which the page identifies as an implementation of FlashAttention v2 | Triton fused-attention tutorial |
| PyTorch-facing interface | Python functions such as flash_attn_func and flash_attn_qkvpacked_func, called on PyTorch tensors |
Official FlashAttention README |
What FlashAttention-2 changes
FlashAttention reduced memory traffic. FlashAttention-2 targets a second limit: how work is divided across the GPU. In the paper, Tri Dao writes: “We propose FlashAttention-2, with better work partitioning to address these issues.” The abstract names those issues as suboptimal partitioning across GPU thread blocks and warps. The paper identifies three core changes.
Free tools Windows power users keep installed
One-click scans. No signup required.
Fewer non-matmul FLOPs
Matrix multiplications run far faster on GPU matrix units than other arithmetic. FlashAttention-2 reduces the non-matmul floating-point operations inside the attention computation, so less of the work falls on the slower path.
#1 Best Overall
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- 0dB technology lets you enjoy light gaming in relative silence
- Dual BIOS switch lets you toggle between Quiet and Performance BIOS profiles
- Dual ball fan bearings last up to twice as long as sleeve bearing designs
Parallelism across thread blocks, even for one head
The paper parallelizes attention across thread blocks even when only a single head is being computed. Work for one head is therefore not confined to one block.
Less communication between warps
The paper reduces inter-warp communication through shared memory, which limits how much warps must exchange intermediate results with each other.
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5070 Ti
- Integrated with 16GB GDDR7 256bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
Calling FlashAttention-2 from PyTorch
The PyTorch path is the FlashAttention package’s functions. Work through these steps in order.
- Install for your platform. Follow the installation section of the official README for your CUDA or ROCm environment. Record your PyTorch, CUDA or ROCm, and package versions together, because a combination that works on one machine may not carry over to another.
- Prepare the tensors. Place query, key and value tensors on the GPU, in a half-precision dtype that your installed version accepts (the example below uses float16), with shape (batch, seqlen, nheads, headdim).
- Call the function. Use
flash_attn_funcfor separate q, k and v tensors, orflash_attn_qkvpacked_funcwhen they are packed into one tensor. Options such ascausal=Trueare passed as keyword arguments. - Check the output. Compare the result against a plain PyTorch reference on a small input, using a tolerance suited to your dtype.
import torch
from flash_attn import flash_attn_func
# shape: (batch, seqlen, nheads, headdim); half precision on a supported GPU
q = torch.randn(2, 1024, 8, 64, device='cuda', dtype=torch.float16)
k = torch.randn(2, 1024, 8, 64, device='cuda', dtype=torch.float16)
v = torch.randn(2, 1024, 8, 64, device='cuda', dtype=torch.float16)
out = flash_attn_func(q, k, v, causal=True)
print(out.shape) # torch.Size([2, 1024, 8, 64])The README documents these options: causal attention, local windows, dropout and ALiBi. Availability varies by backend and implementation path, so confirm each option for your GPU and platform before relying on it in a model.
Rank #3
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5060
- Integrated with 8GB GDDR7 128bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
Reading the Triton tutorial
The tutorial page describes its code as “This is a Triton implementation of the Flash Attention v2 algorithm from Tri Dao.” It includes forward and backward paths and benchmark tables. It is written to be studied and modified, not imported as the library API.
The page is maintained on the main branch, so its code and tables can change. Read the version that matches the Triton release you have installed. The tables reflect the hardware and settings shown on the page; measure on your own GPU before drawing conclusions. A practical reading order:
Rank #4
- Powered by Radeon RX 9070 XT
- WINDFORCE Cooling System
- Hawk Fan
- Server-grade Thermal Conductive Gel
- RGB Lighting
- Read the paper’s abstract and its section on work partitioning to see the three changes in context.
- Read the forward kernel in the tutorial and map its program blocks to the partitioning the paper describes.
- Read the backward path and compare how it is organized with the forward path.
- Modify one piece, such as a mask, and validate the result against a reference implementation on small inputs.
Choosing a path
| Goal | Use | Trade-off |
|---|---|---|
| Run attention inside a PyTorch model | FlashAttention package functions | Installation and feature support depend on your GPU and backend |
| Understand how the kernel is built | Triton fused-attention tutorial | Tutorial code, not a stable library API, and it changes with the main branch |
| Experiment with a modified attention kernel | Start from the tutorial and validate against a reference | You own correctness, numerical behavior and performance testing |
Neither path is universally faster. The choice depends on whether you need a callable function or kernel source you can change.
Reading the benchmark numbers
The paper’s experiments used an A100 80GB SXM4, sequence lengths from 512 to 16k, a hidden dimension of 2048, and head dimensions of 64 or 128. The figures below are the paper’s reported results for that setup, from 2023. They are not a current leaderboard.
Best Value
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- Phase-change GPU thermal pad helps ensure optimal heat transfer, lowering GPU temperatures for enhanced performance and reliability
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- Dual-ball fan bearings last up to twice as long as standard conventional sleeve bearings designs
- 0dB technology lets you enjoy light gaming in relative silence
| Reported figure | What it measures | Comparison |
|---|---|---|
| 1.3–2.5× faster | Attention in the paper’s evaluated comparisons | FlashAttention-2 versus FlashAttention in Triton |
| About 1.3–1.5× in forward and about 2× in backward | Forward and backward passes reported separately | FlashAttention-2 versus FlashAttention in Triton |
| Up to 10× faster | Attention in the paper’s evaluated comparisons | FlashAttention-2 versus a standard attention implementation in PyTorch |
| Up to 230 TFLOPs/s, reported as 73% of theoretical maximum on A100 | Attention kernel throughput | FlashAttention-2 on A100 |
| Up to 225 TFLOPs/s and 72% model FLOPs utilization per A100 | End-to-end training in the paper’s reported experiments | Training runs, not a single attention kernel |
Kernel throughput and end-to-end training are different measurements, so do not treat the 230 TFLOPs/s figure and the 225 TFLOPs/s figure as the same claim. The A100 results do not predict performance on other GPUs. Source: FlashAttention-2 paper (PDF).
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Hardware and platform support
| Platform | Examples listed in the README | Practical note |
|---|---|---|
| NVIDIA Ampere | A100, RTX 3090 | Confirm the feature list for your specific card |
| NVIDIA Ada | RTX 4090 | Confirm the feature list for your specific card |
| NVIDIA Hopper | H100 | Confirm the feature list for your specific card |
| AMD ROCm | Composable Kernel and Triton backends | The backend determines which paths and features are available |
A listed family is not a guarantee that every feature behaves identically on every device. Check the current README for your architecture before installing.
Quick Recap
Troubleshooting checklist
- Install or import failure: confirm that the CUDA or ROCm toolchain matches your PyTorch build, then follow the README installation path for that platform.
- Unsupported GPU: confirm that your architecture falls within a listed family for your platform.
- Feature error: remove the option, or move to a backend that supports it, using the option notes above.
- Output mismatch: check the dtype and the (batch, seqlen, nheads, headdim) layout first, then compare against a float32 reference with a tolerance suited to half precision.
- Tutorial code that differs from this article: the tutorial tracks the main branch, so match it to the Triton version you installed.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →
