Meta’s TLX-based Jagged Flash Attention (JFA) kernel is designed for packed variable-length sequences on NVIDIA Blackwell, and its authors report better performance than the May 2026 FlashAttention-4 (FA4) implementation on their tested jagged workloads. The reported advantage is workload-specific: it varies by forward or backward pass and does not establish a universal win across attention shapes or Blackwell GPUs.
What Jagged Flash Attention is designed to do
Jagged attention processes variable-length sequences as packed data rather than padding every sequence to the same length. JFA takes concatenated Q, K, and V tensors plus offsets marking sequence boundaries. Its public API documents query and key/value offsets as prefix sums, each with one more element than the batch size. This representation avoids doing attention work on padding tokens.
As an Amazon Associate I earn from qualifying purchases.
The October 1, 2026 PyTorch Blog article by Han Xu, Jacky Zhou, Jackie (Jiaqi) Xu, Hongtao Yu, Peng Chen, Darren Liu, Dev Shankar, Max Leung, Nick Riasanovsky, Hao Yan, Manman Ren, and Yuanwei (Kevin) Fang places the kernel in Meta’s GEM training context. The authors say padding can waste up to 50% of compute in that context, citing an earlier GEM training-system explanation; this is not a general estimate for all variable-length workloads.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why the authors moved the kernel to TLX
The authors describe their earlier Triton baseline as leaving much of pipeline depth, on-chip data movement, and scheduling to the compiler. TLX gives them more explicit control over hardware-aware memory use, asynchronous operations, barriers, and warp specialization. That control matters when the work per sequence varies: the kernel must move data and perform attention while avoiding idle compute resources.
#1 Best Overall
- ATTENTION BUYERS: EXPORTING this item outside the U.S. requires strict compliance with the Bureau of Industry and Security (BIS) U.S. Export Administration Regulations (EAR). Non-compliance carries fines up to $1 MILLION PER VIOLATION AND UP TO 20 YEARS IMPRISONMENT for both the buyer and seller. To uphold our regulatory obligations, we strictly follow BIS "Know Your Customer" guidance by screening each order and immediately aborting transactions showing unresolved regulatory red flags. ***
- [Double-Flow-Through Design] The RTX PRO 6000 Blackwell features a double-flow-through cooling design, optimizing efficiency and airflow to sustain peak performance under 600W power loads. | [5th Gen Tensor Cores] Deliver up to 3X the performance of the previous generation and support for FP4 precision for faster AI model processing times with reduced memory usage, enabling local fine-tuning of LLMs and generative AI | [4th Gen Ray Tracing Cores] Double the ray-triangle intersection rate of the previous generation to create photoreal, physically accurate scenes and immersive 3D designs with RTX Mega Geometry, which enables up to 100X more ray-traced triangles.
- [PCIe Gen 5] Support for PCIe Gen 5 provides double the bandwidth of PCIe Gen 4, improving data-transfer speeds from CPU memory and unlocking faster performance for data-intensive tasks like AI, data science, and 3D modeling. | [GDDR7 Memory] With 96 GB of GPU memory and 1.8 TB ps bandwidth, it can tackle massive 3D and AI projects, fine-tune AI models locally, explore large-scale VR environments, and drive larger multi-app workflows.
- [DisplayPort 2.1] Achieve unparalleled visual clarity and performance, driving high resolution displays at up to 8K at 240 Hz and 16K at 60 Hz. Increased bandwidth enables seamless multi-monitor setups while HDR and higher color depth support ensures superior color accuracy for precision work, such as video editing, 3D design, and live broadcasting.
- [NVIDIA Blackwell Streaming Multiprocessor] The new SM features increased processing throughput, and new neural shaders that integrate neural networks inside of programmable shaders | DLSS 4: Multi Frame Generation ensures ultra-smooth frame pacing for lifelike simulations. | [Universal MIG] Divide a single RTX PRO 6000 Blackwell into multiple isolated instances, each with dedicated resources, allowing for concurrent execution of multiple workloads, optimized GPU utilization, and secure isolation of different applications or users. [WARRANTY] 3 YR Manufacturer's Warranty. Bulk OEM Packaging. Retail Packaging is NOT included.
The PyTorch Blog article frames the motivation bluntly: “Attention is the single slowest kernel in GEM.” That is the author team’s characterization of its GEM workload, not a claim about every model or training system.
How the kernel schedules jagged work
In an equal-length batch, similarly sized tiles can be distributed predictably. In a jagged batch, sequence lengths differ, so tiles create uneven amounts of work across streaming multiprocessors (SMs). The kernel uses a persistent, warp-specialized structure: separate warp roles and asynchronous pipelines let data loading, softmax operations, and matrix multiplication overlap rather than proceed as a single serial sequence.
- Explicit memory and synchronization: The implementation allocates shared memory and tensor memory deliberately and uses barriers to coordinate asynchronous TMA loads and MMA operations.
- Work distribution: Tile balancing and Cluster Launch Control (CLC) help distribute uneven jagged work across SMs.
- Pipeline and resource tuning: The authors describe staging dQ work, releasing tensor memory earlier, and peeling loops to improve execution and resource use.
- Collaborative matrix operations: The specialized backward path adopts FA4’s two-CTA collaborative MMA design for a constrained broadcast-query case.
These are implementation choices described by the authors, not a guarantee that every jagged attention kernel will benefit from the same scheduling strategy.
Recommended Free Tools
Why broadcast queries complicate backward
In the production-style Hierarchical Seed Pooling (PMA) case described in the article, a dense query is broadcast across jagged sequences. In the backward pass, the gradient for that shared query must accumulate contributions from the batch. That cross-program reduction is different from computing an independent query gradient for each sequence.
Rank #2
- Professional GPU with Blackwell Architecture
- Blackwell Architecture
- 24GB GDDR7 with PCIe 5.0 & Ray Tracing
- AI Workstation
The authors report that their two-CTA method improves backward throughput by about 12%—or lowers latency by about 11%—for the production broadcast-query, head-dimension-128 case compared with their single-CTA path. This is a scoped ablation of the kernel design, separate from the broader comparison against FA4.
What the FA4 comparison measured
The PyTorch Blog authors compared their TLX kernel with the May 2026 FA4 implementation on an NVIDIA B200 using BF16. They report results for two different regimes: production-style jagged shapes with broadcast queries, and a separate equal-length, LLM-style dense workload. Their optimization and latency work used Nsight Compute hardware counters, ptxas spill information, and TritonBench profiler-measure runs. The figures below are the authors’ reported results; they are not an independent reproduction.
| Workload and measure | Reported result | How to read it |
|---|---|---|
| Tested jagged shapes, forward, BF16 on B200 versus May 2026 FA4 | About 13% better average forward performance | The average is for the authors’ tested shape mix. They say TLX trails on the longest sequences at high density. |
| Tested jagged shapes, backward, BF16 on B200 versus May 2026 FA4 | About 50% better average backward performance | The authors report faster backward performance across their tested jagged shapes. |
| Reported dense LLM-style shapes, forward, BF16 on B200 versus May 2026 FA4 | About 87% of FA4’s forward performance | This is the separate equal-length dense regime, not the jagged production result. |
| Reported dense LLM-style shapes, backward, BF16 on B200 versus May 2026 FA4 | About 17% better backward performance | This dense-workload result should not be conflated with the approximately 50% jagged backward average. |
The authors also sweep sparsity from highly variable sequence lengths toward nearly uniform lengths. That matters because “jagged” covers a range of shape distributions: results on one mix do not predict results on another, and the long-sequence, high-density forward case is already a reported exception to the average.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
For context only, the FA4 paper by Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao reports up to 1.3× speedup over cuDNN 9.13 and 2.7× over Triton on B200 BF16, reaching 1,613 TFLOPs/s (about 71% utilization) under that paper’s own benchmark settings. Those figures describe FA4’s separate benchmarks; they do not validate or reproduce the JFA-versus-FA4 comparison.
Rank #3
- Form Factor: Plug-in Card
- Cooler Type: Active Cooler
- Maximum Power Consumption: 70W
- Length: 6.6
- Height: 2.7
What the published package supports—and where specialization stops
The facebookresearch/ads_model_kernel_library repository’s tlx_jfa README documents jagged self-attention and cross-attention, PMA or broadcast-query attention, symmetric sliding windows, grouped-query attention in forward, and autograd backward. Its optimized two-CTA backward is narrower than that feature list: it is documented for broadcast-query PMA with head dimension 128, one query group, no sliding window, and load balancing enabled. Other shapes use a general one-CTA backward path.
- The package requires a Blackwell SM100-or-newer GPU and supports head dimensions up to 128.
- The README lists Python 3.12 as an available setup path and specifies
fbtriton==3.6.1. - It gives a PyTorch CUDA 12.8 wheel as an installation example; documentation alone does not establish that a particular environment will install or run successfully.
MXFP8 and block-sparse experiments
MXFP8
The article describes an MXFP8 variant that uses E4M3 values, E8M0 block scales, and TLX block-scaled MMA while reusing the kernel skeleton. The authors report forward performance above FA4’s FP8 kernel and backward parity with FA4 at dense. These are claims from the article, not independently reproduced results, and should be read within the authors’ tested setup.
Block-sparse attention
The block-sparse design first uses a scoring kernel to pool query and key blocks and select top-k key/value blocks; the attention kernel then iterates over those selected blocks. The article says this variant supports broadcast queries, grouped-query attention, and windowing. At a 0.5 selection ratio, the authors report roughly 1.3–1.5× forward speedup over dense attention for the sequence lengths they tested. That result is limited to those tested lengths and is not a general guarantee for sparse attention. The experiment is described in the article; it is distinct from the feature list in the repository README.
What the code-size comparison does and does not say
The authors estimate their TLX kernel at about 3.2K lines, compared with roughly 10K lines for FA4 CuteDSL kernels. They offer this as a code-size and maintainability point. Line counts alone do not independently establish engineering effort, ease of maintenance, or performance portability.
How to judge whether the result applies to your workload
- Sequence structure: Compare variable-length packed sequences with equal-length batches, and check whether sequence lengths are highly uneven or nearly uniform.
- Query layout: A broadcast query has a different backward reduction requirement from per-sequence queries.
- Pass and dtype: Forward and backward results differ, as do BF16 and the article’s MXFP8 experiment.
- Shape and feature constraints: Head dimension, query groups, sliding windows, and load balancing can determine whether the specialized two-CTA backward applies.
- Hardware and comparator: The headline figures are for B200 and the May 2026 FA4 implementation; the package documentation requires Blackwell SM100+.
The practical conclusion is not that TLX JFA replaces FA4 everywhere. It is that explicit scheduling and memory-pipeline control can pay off on the authors’ tested production-style jagged workloads, especially in backward, while forward performance depends more visibly on sequence shape and density.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
