Free tools Windows power users keep installed
One-click scans. No signup required.
A custom ignore_index can silently remove the end-of-sequence (EOS) token from training if its integer ID is also used as the loss function’s ignore value. That is what Panagiotis (Panos) Gkilis reports in an autoregressive audio model: the run stayed finite and appeared healthy, but its loss did not supervise EOS targets. His case also shows why a falling loss alone cannot establish that a model has learned when to stop.
How one integer made EOS invisible to the loss
In Gkilis’s reported setup, audio tokens occupied IDs 0 through 1023, and EOS was assigned ID 1024. The autoregressive output layer therefore had 1025 classes:
As an Amazon Associate I earn from qualifying purchases.
logits = nn.Linear(d_model, NUM_AUDIO_TOKENS + 1)(hidden_states)
loss = F.cross_entropy(
logits,
targets,
ignore_index=NUM_AUDIO_TOKENS,
)With NUM_AUDIO_TOKENS set to 1024, the output classes were 0–1024, while ignore_index was also 1024. In PyTorch cross-entropy, target positions equal to the ignore value are excluded from the loss. Since 1024 was a real EOS class in this output layer, every EOS target at those positions was omitted from supervision.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →This is a collision between a valid target ID and a loss sentinel, not an error that necessarily makes training crash. The model can still receive loss from its other target positions, producing finite and plausible aggregate values. Those values say nothing about the omitted EOS positions.
#1 Best Overall
Why the same sentinel can be safe in one stage and wrong in another
An integer’s meaning depends on the output classes at the stage where the loss is calculated. Gkilis reports a non-autoregressive stage with 1024 classes, where 1024 was outside the valid class range, and an autoregressive stage with 1025 classes, where it was the valid EOS class. Checking the sentinel or output width in isolation would miss that distinction.
For each loss calculation, compare the ignore value with the actual valid target range. If the output has classes 0 through C−1, an ignore value must not equal any class that can appear as a supervised target. In a model that adds EOS to C ordinary tokens, the output width is C+1 and EOS may be class C.
What the reported run showed—and what it does not prove
Gkilis reports two different comparisons. In a minimal two-arm reproduction, the broken arm finished at loss 0.0035 and the fixed arm at 0.0034. Those close values illustrate that the aggregate loss did not clearly expose the EOS exclusion; they do not establish a general expected loss difference or prove that all training runs with this bug behave alike.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsIn a separately reported 200-epoch run evaluated every 50 epochs on an utterance-held-out split of 32 examples, the epoch-100 checkpoint had mean P(stop) of 0.4655 and EOS was the top-ranked class on 18 of 32 examples. At epoch 150, training loss was 20% lower, while mean P(stop) had fallen to 0.2159—a 54% decline—and EOS was top-ranked on 8 of 32 examples. This is a reported experiment, not independently validated evidence that lower loss generally worsens stopping.
Rank #3
After correction, the article reports mean P(stop) of 0.4655, EOS as the top-ranked class on 18 of 32 held-out examples, and an end-to-end synthesis that stopped at frame 203 with a 350-frame ceiling. The author also reports that P(stop) plateaued around 0.35–0.47 despite trying two learning rates and increasing data from 224 to 1313 utterances, a 5.9-fold increase; the cause was not confirmed. These are the author’s reported measurements, not independent validation.
Checks to catch this before trusting a run
- Compare class range and sentinel at every stage. Record the output width, valid target IDs, and
ignore_indexfor each loss. Flag a configuration when the ignore value is also a valid output class. - Inspect targets after tokenization and collation. Confirm EOS appears where expected in the actual batches and has not been converted to the ignore value or otherwise removed before loss calculation.
- Verify which classes reach the loss. Track target-class counts at loss input, including ignored positions. Gkilis proposes flagging classes that never appear as positive targets during an initial epoch, while gating a dead-class alert on broad class coverage so short or sparse runs do not trigger the same warning as a structural exclusion.
- Evaluate stopping directly. On terminal frames, measure P(stop) and EOS rank or argmax; also test whether generation terminates autonomously rather than only reaching a configured maximum length.
- Choose checkpoints using task behavior as well as loss. A checkpoint with lower aggregate loss need not be better at stopping. Keep the metric aligned with the capability the model must demonstrate.
Reading the evidence in context
Gkilis, founder of BedVibe Studios, concludes: “The training loss is not a sufficient statistic for model capability.” In this case, the reason is concrete: the loss calculation excluded a behaviorally important target, so the reported scalar did not measure how well the model learned that target. The conclusion is useful beyond EOS, but the particular measurements and outcomes above belong to the author’s reported model and experiments.
Quick Recap
Rank #4
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
