The strongest example I had used to show my ML linter catching a real training failure did not support the claim I made about it. Two independent audits found that the Coqui XTTS log I shipped stopped at step 72,900, even though the run continued to step 125,039. The complete record changed the interpretation: held-out evaluations improved, while one training-loss series still ended above its own minimum. That is not enough to say whether the model’s quality worsened or improved.
The correction left trainproof’s verdict at FAIL, but exposed why a deterministic result is not the same as a validated conclusion about model quality.
As an Amazon Associate I earn from qualifying purchases.
What the audit found in the XTTS run
Before release 0.22.0, I presented a Coqui XTTS fine-tune as the one failure in my validation gallery that had not been deliberately injected. The evidence was incomplete: the shipped log was a prefix, not the full run. The correction and comparison are documented in the trainproof repository.
| Evidence | Shipped log prefix | Complete run |
|---|---|---|
| Training steps represented | Ended at step 72,900 | Continued through step 125,039 |
| Last reported best-model checkpoint | best_model_49880.pth |
best_model_124700.pth, promoted at step 124,700 |
| Retained held-out evaluations | Three visible | Six, improving from 4.8813 to 2.5894 |
The prefix’s last BEST MODEL line named best_model_49880.pth. Treating that as evidence that the run had “diverged on its own” relied on a record that ended long before the trainer’s later checkpoint promotion and evaluations. The complete log undercuts the claim about the run as a whole.
#1 Best Overall
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
What the measurements do—and do not—establish
The corrected claim is narrower. As maintainer Panagiotis Panos Gkilis put it in the repository correction: “What the retained evidence supports is narrower than the original claim: the per-micro-batch training display loss ended above its own minimum.” That describes a measured training series, not the quality of the resulting model.
The held-out evaluations improved across the six retained observations, but loss movement alone is not a perceptual assessment. No audio, mean opinion score (MOS), or other perceptual evaluation was retained for this run. The evidence therefore establishes neither that the audio quality worsened nor that it improved.
Rank #2
Why trainproof still returned FAIL
The repository’s explanation is in its correction and rule documentation: TP-DIVERGE evaluates one training series, and the reader resolves competing series by density. In this event file, that meant selecting 1,251 per-micro-batch points and discarding a five-point epoch-aggregated series. The selected dense series ended at a loss 1.88 times its minimum; the discarded epoch series was at its minimum. The improving held-out curve did not override that selection.
Recommended Free Tools
Changing a threshold just to make this particular run pass would have tuned the rule to a preferred answer. I left the result at FAIL and documented the reader’s limitation. Release 0.22.0 did not fix the single-series behavior. That verdict says how the rule classified the selected measurements; it does not prove that the run’s model failed at its task.
What else changed in the audit
The correction covered more than the headline example. The release notes describe these changes and remaining caveats:
- Missing optional dependency: When
sentencepiecewas absent,trainproof tokenizerpreviously exited 1 with FAIL. It now reports NOT-CHECKED and exits 2, distinguishing an unavailable check from a fault in the user’s run. - Data-loader side effect: The Hugging Face callback’s label inspection at
on_train_beginopened a fresh iterator over training data. With a map-style loader and random sampler backed by a generator, that could consume generator state and shift batch order. The callback’sobjective_checknow defaults to false. - Evidence wording: Four evidence strings were corrected because they did not accurately describe the measured value. One message said that 100.0% of steps had a zero learning rate even though the values were all
-1e-4. - Messages that overstate mechanisms: The repository says two independent audits also found messages that claim mechanisms the log cannot establish. For example,
TP-NAN-GRADsays non-finite gradients “reached the optimizer,” which may be untrue if GradScaler skipped a step. Those messages were not changed in 0.22.0; the repository describes their correction as architectural work.
The maintainer reports 497 tests for release 0.22.0, with a test for each correction that failed against the pre-repair code. That is the author’s reported test count, not an independent audit of the test suite. The repository says the release changed no rule ID, threshold, or detection predicate.
Rank #4
What a deterministic linter can tell you
Trainproof describes itself as a deterministic linter for training runs, installable with pip install trainproof. It applies rules to logs and prints evidence; a check that cannot run should be NOT-CHECKED rather than presented as passed. The tool’s mechanical findings are separate from the question of whether corrupted data or another issue caused a model to learn something useful.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →That distinction matters because neither repeatability nor precise-looking numbers establish that a rule is well calibrated. The repository says the rules have not been tested against a population with independently known outcomes, so their false-positive rate is not established. The injected-fault gallery is a regression suite, not a representative sample from which to infer real-world accuracy.
Best Value
In a separate author-reported validation gallery, 18 Qwen2.5-3B QLoRA runs covered six configurations at three seeds. One shuffled-label run reduced its loss by 62% while learning no useful mapping; the author says a single-run loss curve did not reveal this and comparison with a known-good baseline exposed the problem. This controlled experiment illustrates a limitation of loss curves, not an independently calibrated guarantee for trainproof. More context appears in the author’s BedVibe Studios Engineering article.
Read the evidence before accepting the message
In the repository correction, Gkilis summarizes the lesson: “Read every finding’s evidence string as the measurement, and its message as an interpretation that may exceed it.” For users, that means checking which series a rule selected, what the log actually contains, and whether a message describes an observation or infers a cause. For maintainers, it means treating a convincing example as provisional until the underlying record and the rule’s behavior have been checked.
The XTTS correction did not show that the linter was useless or that the model succeeded. It showed that my best example could not sustain its original conclusion. The surviving finding is about a training-loss series; the broader question of model quality remains unanswered by the retained evidence.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
