The author did not fix the bug in laya’s multilingual checkpoint. They showed that its ordinal score predictions almost never select the first-listed option, traced the effect to slot position rather than to a Japanese label or a faulty harness, and contributed a narrow regression check (GitHub PR #259) that the maintainer merged. The model itself still needed retraining, and the records available for this account, dated September 22 to 24, 2026, show that retraining as pending. The 20k-star figure comes from the title; star counts change over time.
What laya is and what was being tested
laya is a non-autoregressive model that the project describes as a “System 1” decision model. You give it a text and one or more typed questions: choice (pick one of several labels), score (pick a point on an ordinal scale), or a yes/no question. It returns answers and probabilities in a single forward pass instead of generating a text reply. The project publishes English and multilingual checkpoints.
As an Amazon Associate I earn from qualifying purchases.
The experiment began as a Japanese-language baseline for a separate project. The author wrote 300 Japanese business emails and 290 English ones. Labels were fixed first, then a local language model wrote an email to match each set of labels, and any email containing a label word was rejected and regenerated. Every email carried three questions: a department choice, an ordinal urgency score, and a cancellation-intent yes/no. These are the author’s own synthetic benchmark data, not a published or representative dataset, so the accuracy figures describe this benchmark and should not be read as general performance.
The Japanese baseline looked reasonable on two tasks and poor on the third. Department choice reached 0.747 accuracy against a 0.380 majority-class baseline. The yes/no task reached 0.543 accuracy against a 0.703 majority baseline, with an AUROC of 0.523. The urgency score was the problem: its RPS was 0.232 against a 0.197 majority baseline (lower is better). The author reports that the lowest urgency option, “not urgent,” was the correct label for 77 of the 300 emails and was never predicted.
#1 Best Overall
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
The bug: the first-listed option disappears
To test whether the missing label was a quirk of one wording, the author ran five schema conditions: the original order, a reversed order, reworded options, reworded and reversed options, and a four-level scale. In the original and reversed schemas, “not urgent” was chosen 0 times when it was listed first and 250 times when it was listed last.
Across all five Japanese conditions, the first slot was selected 0 or 1 times out of 300. The same five conditions were then run on 290 English emails. The table shows first-slot selections for both checkpoints. Each cell is a count of items where the first-listed option was chosen.
| Checkpoint and language | Original | Reversed | Reworded | Reworded and reversed | Four levels |
|---|---|---|---|---|---|
| laya-multilingual, Japanese (n=300) | 0 | 0 | 1 | 1 | 0 |
| laya-multilingual, English (n=290) | 0 | 0 | 0 | 0 | 0 |
| English laya, Japanese (n=300) | 13 | 56 | 8 | 1 | 110 |
| English laya, English (n=290) | 65 | 74 | 0 | 5 | 4 |
The English checkpoint does select the first slot, sometimes often. It is the multilingual checkpoint that never does. The first-slot rate for English laya on English emails in the original order was 65 of 290 (about 22%), and 74 of 290 (about 26%) in the reversed order. The GitHub issue opened on September 22, 2026 (issue #131) reports that range, and it applies only to those two original orderings. The full table above is the accurate picture.
The setup in that issue used laya 0.3.4 through the README’s laya.load() and agent.predict() calls. On the English emails, the multilingual checkpoint’s score RPS was 0.340 against a random baseline of 0.197, and the English bool AUROC was 0.355.
Ruling out a Japanese-language cause
The English run was the first test of whether Japanese text caused the problem. The multilingual checkpoint showed the same zero-selection pattern on English, so language alone does not explain it. The effect also cannot be a Japanese-specific problem in the score head, because the English checkpoint behaves differently on identical English items.
Rank #2
Position or label? The per-item shuffle
The obvious objection is that a single label, such as “not urgent,” might be suppressed wherever it appears. The author addressed this by shuffling the options independently for each email, so that each label could appear in every slot. In that Japanese run:
- The first slot was selected 0 times out of 300.
- Slots two and three received 149 and 151 selections.
- The three labels received 75, 93, and 132 selections, and each label occupied the first slot on 90, 109, or 101 items.
The missing selections followed the first slot, not any particular label. The last slot showed no comparable preference. The question the author kept returning to, “Is it the position or the word?”, resolved toward position.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Checking the harness and the checkpoint
A custom harness can produce a bias of its own, and the reader’s natural suspicion is that the measurement is wrong. The author’s write-up credits a third party, AlKor13, with examining raw marker logits and testing three identical options. As the author reports it, switching only the checkpoint made the position effect appear or disappear. The multilingual checkpoint showed a strong slot effect even when all three options had identical text.
AlKor13 also found that removing the level N: prefix from each option removed the slot-0 suppression in raw logits. The author notes that this changed the input into a format the model was not trained on, so it does not prove the prefix can be dropped as a remedy. These controls are described as reported by the author; they are not independently reproduced in the sources available here.
Why rewording was not called a fix
The author compared three renderings of the score options: the shipped level N: format, a version without that prefix, and word ordinals. Each was tested on identical examples with paired comparisons. Two of the four language-and-rendering comparisons were significant (McNemar p-values of 0.0007 and 0.0003), and two were not (0.145 and 0.350). Different renderings helped different language conditions.
Changing the rendering reshuffled individual predictions far more than the headline numbers suggest. Under the prefix-free rendering, 56.7% of Japanese items and 56.9% of English items changed correctness. Headline accuracy moved by 6.6 points on the Japanese set and 16.2 points on the English set. In one control, removing the prefix lowered English-checkpoint accuracy from 0.583 to 0.500.
The author’s conclusion is that the rendering behaved unstably and depended on the checkpoint. “Drop the prefix and it is fixed” was not supported by these results.
The contribution: PR #259
The merged pull request, PR #259 in the laya repository, adds research/eval/presentation_checks.py and offline regression tests. It does not change any model weights or any code under laya/, and it adds no dependencies. It contains two checks.
Check 1: the slot-0 logit metric
When options have identical text, the check compares the raw slot-0 marker logit with the mean across slots. A strongly negative value means the first slot is suppressed. The threshold is at least −0.20.
Check 2: the first-slot rate
The script takes three real levels, presents them in all six permutations for each of ten fixed English support messages, and measures the proportion of decisions that select the first slot. The threshold is at least 0.15.
Rank #4
Before running either check, the script compares its own inference path with Agent.system_one. A failed gate and a harness mismatch return different exit codes, so a failure is not mistaken for a broken setup.
Results on the documented setup
The PR reports one run on the documented CPU, fp32, with laya 0.3.7:
| Checkpoint | Slot-0 logit metric (threshold ≥ −0.20) | First-slot rate (threshold ≥ 0.15) | Result |
|---|---|---|---|
| English checkpoint | +0.664 | 0.217 | Passes both gates |
| Multilingual checkpoint | −0.492 | 0.017 | Fails both gates |
The PR also reports that probability outputs matched the package path to within about 4.98e-5 and 4.92e-5 (maximum differences). These figures apply to this setup and checkpoint version; they are not a claim about every runtime.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What the check does and does not establish
The PR is explicit about its limits. It uses ten short English messages, English only, the score question type only, and thresholds calibrated on CPU fp32. Passing the gate is not the same as being accurate. A checkpoint can pass both gates and still make poor urgency calls, and the check says nothing about whether the model’s urgency predictions are correct. It only tests the targeted positional behavior.
Free tools Windows power users keep installed
One-click scans. No signup required.
The multilingual checkpoint fails the check, and the issue stays open. The maintainer’s position, from the PR discussion, is that the remaining work is a position-balanced multilingual checkpoint, which requires retraining. The check is designed to help evaluate that retrain.
Best Value
The maintainer, NandhaKishorM, responded in the PR:
“Thank you, a label-free check that answers one question (did the retrain remove the score slot prior?) is exactly what #131 needs, and exit codes that separate a failed check from a harness disagreement make it easy to trust. Merging.”
Working inside a repository you do not maintain
The author describes several scope decisions that shaped the contribution. The regression tests are standalone scripts rather than pytest tests. Model checkpoints are not loaded in CI. A separate discussion about wiring research tests into the project was unresolved, so the PR does not register its offline tests in CI. Another contributor was already building a broader option-permutation framework, so the author kept this check focused on issue #131 rather than duplicating that work.
Recommended Free Tools
The write-up’s own framing of the approach is a one-line summary the author offers, and it fits the tone of the whole project: measure the behavior, make the measurement hard to dismiss, then hand the maintainer a tool they can run.
Applying the method to your own model
- Start by listing which slot, label, and language factors could explain the result, then design a run that changes one factor at a time.
- Shuffle options independently per item so each label can occupy every slot. This separates position from label identity.
- Run the same conditions on a second language or dataset before blaming the language.
- Check your harness against the package’s documented inference path, and confirm that probability outputs agree within a tolerance you state.
- When you change a rendering or prompt format, compare paired outcomes per item, not only the aggregate. Report how many items flipped.
- Wrap the check so a failed gate and a broken harness return different exit codes.
- State what the check cannot tell you, then leave the retraining decision to the maintainer.
The author’s result is useful because it is narrow. It proves a specific positional failure exists and gives a reproducible way to see whether a future checkpoint removes it. It does not prove the model is good at urgency triage, and it does not replace the retrain.
Quick Recap
“
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
