A 310-million-parameter Japanese encoder trained on a few hundred labeled examples decisively beat Jev on one long-document topic task—but it did not establish a clear win on either of two polarity tasks. In Ikkun’s 2026 experiment, the result depended on the task: the encoder led by 12 percentage points on livedoor news classification, while Jev and the encoder were statistically tied on Rakuten reviews and ChABSA financial sentences. If you have a few hundred labels, the practical answer is to test your own held-out data, not assume one approach wins across the board.
What did the 750-row comparison test?
Ikkun compared Jev with sbintuitions/modernbert-ja-310m, a 310-million-parameter Japanese encoder fine-tuned with a classification head. The experiment used three Japanese datasets, each reduced to 250 rows:
As an Amazon Associate I earn from qualifying purchases.
- livedoor news: nine-class topic classification; mean document length of 1,174 characters.
- Rakuten reviews: two-class polarity classification; mean length of 138 characters.
- ChABSA: three-class financial-sentence polarity classification; mean length of 92 characters.
The encoder was evaluated with five-fold out-of-fold predictions: each round trained on 200 rows and predicted the remaining 50. Ikkun reports that systems were compared on identical row IDs using McNemar’s paired test. These are the experimenter’s results and protocol, not an independently reproduced benchmark. Ikkun’s Agent Journal report.
Which system won on each task?
| Dataset and task | 310M encoder | Jev | Reported paired result |
|---|---|---|---|
| livedoor, nine-class topic | 88.8% | 76.8% | Encoder ahead by 12.0 percentage points; p=0.00007 |
| Rakuten, two-class review polarity | 92.8% | 94.4% | Jev ahead by 1.6 points; p=0.50, reported as a tie |
| ChABSA, three-class financial polarity | 75.2% | 74.0% | Encoder ahead by 1.2 points; p=0.83, reported as a tie |
The clear result was limited to livedoor: the encoder’s advantage was large in this sample and statistically significant under the author’s paired test. The small numerical differences on the two polarity tasks were not decisive. A nonsignificant result with 250 rows does not prove that the systems are equivalent; it means this experiment did not establish a reliable difference.
#1 Best Overall
A simple text baseline was competitive on topics
A character n-gram plus logistic-regression baseline scored 88.4% on livedoor, 73.6% on Rakuten, and 64.0% on ChABSA. It nearly matched the encoder on livedoor, but scored lower on the polarity datasets. Ikkun tested only one configuration of this baseline, so these figures do not establish its best achievable performance.
Why might a small trained model win on one task but not another?
Ikkun’s interpretation is that topic labels can be strongly signaled by vocabulary. With labeled examples, a model can learn that vocabulary-to-topic relationship without requiring a broad judgment about meaning. Polarity classification, by contrast, asks whether a short text expresses a particular stance or sentiment; in these two tests, fine-tuning did not produce a decisive advantage over Jev.
That is a useful explanation for these results, not a general rule that encoders always win at topics or decision APIs always win at sentiment. The experiment covered three Japanese datasets and specific systems, not other languages, domains, or production data.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →When should you try a trained encoder instead of Jev?
If you have a few hundred labeled examples
Try a supervised model when the label is tied to recognizable vocabulary or recurring domain-specific terms, especially for topic classification. The livedoor result supports testing that route; it does not guarantee the same gain in your dataset. Use held-out examples from your own domain to decide.
If the task is sentiment or another polarity judgment
Do not budget on the assumption that a few hundred labels will beat a decision API. The encoder and Jev were statistical ties on both polarity tasks in this report. Your own text, label definitions, and error costs may produce a different outcome.
If you have no labeled examples
A decision API such as Jev can be a starting point when you need classification without first assembling a labeled training set. The comparison reports Jev version 1.13.0; its service terms and capabilities can change. Treat the experiment’s reported API price—$0.042 per 1 million input tokens, with output free—as a figure reported by Ikkun, not a current quote, and verify current terms with the provider.
Rank #4
- Learning Japanese Workbook for Beginners: Hiragana Katakana And Kanji Quick and Easy Way to Learn the Basic Japanese Upto 300 Pages
- ABIS BOOK
- Independently published
If you use Jev to create training labels
A model trained only on Jev-generated labels can learn Jev’s mistakes as well as its decisions. If the goal is to surpass the teacher, review uncertain cases and add better human or otherwise trusted labels rather than treating every generated label as ground truth.
Recommended Free Tools
How should you make the comparison useful for your own data?
- Define the task and labels. Decide what counts as a correct topic or polarity label, and check that annotators apply the categories consistently.
- Reserve representative evaluation data. Keep a held-out set from the same domain and distribution you expect in use. Do not tune repeatedly against the final evaluation set.
- Compare on identical examples. Run each candidate on the same held-out rows, then inspect both overall accuracy and the kinds of mistakes each makes. Paired comparison matters because the systems may succeed and fail on overlapping examples.
- Include operational constraints. Consider data privacy and network access, latency, inference and labeling costs, confidence calibration, and the consequences of a wrong classification. Accuracy alone does not determine the right deployment choice.
- Plan for unresolved cases. If both approaches plateau, investigate label quality and data coverage, then consider routing low-confidence cases to a stronger model or human review. That is Ikkun’s suggested workflow, not a separately measured result in this experiment.
What limits the reported results?
- Small samples: each task had 250 rows, so the comparison may miss smaller differences.
- Long-document truncation: livedoor inputs were truncated at 512 tokens, which may exclude useful content from longer articles.
- Raw confidence scores: the encoder used raw argmax predictions; its confidence was not temperature-scaled or otherwise calibrated.
- Limited baseline tuning: only one character n-gram configuration was tested.
- Restricted scope: the datasets were Japanese, and the results apply to the reported systems and versions. They do not establish outcomes for other languages, domains, versions, hardware, or production distributions.
For the tested setup, Ikkun reports median per-item latency of 0.10–0.45 seconds for the encoder and 1.9–2.4 seconds for Jev. These are author-measured, setup-specific figures, not service guarantees. The reported experiments ran on a Ryzen 9 7940HS mini-PC with Radeon 780M graphics; training time and inference latency can differ substantially on other hardware and software configurations.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
