Tensor Shapes in ML: What Gets Checked—and What Doesn’t

A tensor’s shape can act like a function contract: [B, T, d] says the code expects batch, sequence, and feature axes. But in common dynamic tensor workflows, operations usually check whether dimensions are mathematically compatible—not whether each axis means what the programmer intended. That gap lets some axis mistakes run without an error.

What a tensor shape tells you—and what it cannot

For a tensor shaped [B, T, d], the dimensions conventionally stand for batch size, sequence length, and feature width. The shape gives the size and order of axes. By itself, it does not attach enforceable names such as “batch” or “time” to those axes.

#	Preview	Product	Price
1		Princeton Architectural Press In the Bookstore: 1000 Piece Puzzle	$19.15	Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

That distinction matters at function boundaries. A function may expect an input shaped [B, T, d], but ordinary tensor code often sees extents and positions rather than semantic labels. Inspecting x.shape or x.size() shows the extents; it does not establish that the first axis really is the batch axis. PyTorch documents the Tensor.shape accessor for retrieving that information.

Why some shape mistakes raise errors and others do not

Tensor operations have compatibility rules, so they do catch some invalid combinations. In PyTorch broadcasting, dimensions are compared from the trailing end. A pair of dimensions is compatible when their sizes match, one size is 1, or one tensor has no corresponding dimension. If dimensions are incompatible, the operation raises an error. See the PyTorch broadcasting semantics documentation.

#1 Best Overall

Princeton Architectural Press In the Bookstore: 1000 Piece Puzzle

• 1,000-PIECE PUZZLE: This colorful, illustrated 1,000-piece adult puzzle offers the perfect challenge for family and friends.
• FUN-FILLED LITERARY REFERENCES: This illustrated bookstore puzzle scene is teeming with entertaining literary references. Discover hidden literary trivia scattered throughout the puzzle as you piece it together!
• HIGH QUALITY MATERIALS: The attractive and sturdy matte-finished storage box measures 8.375 x 11.375 x 2.5. Quality control checks during production ensure there is virtually no puzzle dust.
• PUZZLE IMAGE ON THE BOX AND INCLUDED IN THE PACKAGE: The box includes the full finished puzzle picture on the front and a separate complete puzzle image insert for two handy references. The final puzzle measures 20 x 25".
• GREAT FOR GIFTING: The perfect gift for bookworms, writers, poets, readers, teachers, librarians, and anyone who has ever felt the thrill of stepping inside a beloved bookstore or getting lost in a great book.

$19.15

Buy on Amazon

Compatibility is not proof that the axes mean the right thing. For example, adding a tensor shaped [B, d] to one shaped [d] can intentionally broadcast the latter across the batch. But if a tensor has an unintended axis arrangement that still satisfies the broadcasting rules, the operation can produce a result rather than flagging the semantic mismatch. Equal or singleton dimensions can make such mistakes especially difficult to spot.

How to make shape assumptions easier to catch

Annotate important function boundaries

Write expected axes where tensors enter and leave important functions, and use a shape-aware annotation or checking tool where it fits your stack. The original article recommends jaxtyping with beartype as one example; its author’s summary is: “The check is yours to write.” That is a warning about ordinary application code, not a claim that no tools can express or validate shapes. Check the tools’ current compatibility and API guidance before adopting them.

Test with unequal axis sizes

Choose test inputs whose potentially confusable axes have different extents—for example, B=3, T=5, and d=7. If batch and sequence lengths are both 4 in a test, swapping those axes may leave the shape looking plausible. Distinct values make many axis-order mistakes easier to expose.

Keep sequence masks and padding assumptions explicit

In sequence pipelines, distinguish real tokens from padding with the relevant mask. For pooling or selecting a last valid token, derive valid positions from that mask rather than assuming a fixed endpoint when padding sides may vary. Ensure that serving-time padding conventions agree with the assumptions in the code; this is a practical safeguard, not a universal prescription for every architecture.

Inspect shapes while debugging

Print or inspect .shape (or .size() in PyTorch) around the operation where dimensions first diverge from expectations. This quickly reveals extents and rank changes. Pair that inspection with assertions, annotations, or tests when the important question is what an axis represents.

Shape checking exists at several different layers

Shape support is not one interchangeable feature. Runtime annotation checks, Python type analysis, and graph or compiler validation operate at different stages and may express different constraints.

These examples qualify the broadest reading of “nobody checks them”: some systems can represent or check shape information. In a typical dynamic workflow, however, the framework’s compatibility rules may not encode the semantic intent behind your axis names.

Quick Recap

Bestseller No. 1

Princeton Architectural Press In the Bookstore: 1000 Piece Puzzle

$19.15

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.