UMKM-Bench is a small, author-built test of whether language models can understand informal Indonesian shop messages and answer from a shop’s own catalog and policies. Its results suggest that typo-heavy slang affected some models more than others, while a short instruction not to guess improved some reported trap-question results. The benchmark is a focused demonstration—not a representative test of Indonesian customer support or a production-readiness ranking.
What UMKM-Bench tests
In a September 25, 2026 DEV Community article, author Rizky Nanda Pratitia describes a benchmark built around three fictional businesses: Kopi Lereng in Sleman, Yogyakarta; Sekar Hijab in Bandung; and Dapur Bu Tini in Semarang. Each has a knowledge base with product catalog details, shipping tables, and policies.
As an Amazon Associate I earn from qualifying purchases.
The author wrote 84 customer messages. Each appeared in formal Indonesian, manually written slang, and a typo-noisy version. The scenarios cover practical shop questions about prices, stock, shipping, cash on delivery, and opening hours, as well as questions whose answers are absent from the shop data. After an initial pilot, the author added 24 harder messages involving calculations, misleading context, a fake discount, and a prompt injection.
Recommended Free Tools
These are benchmark prompts, not verified messages from real shops. Examples include “kak arabika yg setengah kilo ready gk?” (“Is the half-kilo Arabica available?”) and “ongkir ke bpp brpmin” (“How much is shipping to BPP?”). The test notes that “yg” abbreviates “yang,” “gk” means “nggak,” and “bpp” refers to Balikpapan in one example. That abbreviation can be ambiguous: one reported failure treated “bpp” as a district in Yogyakarta rather than Balikpapan. Another example uses “wingi,” a regional word meaning “yesterday.”
#1 Best Overall
For each message, a model was asked to act as a shop administrator and return structured fields: intent, product or SKU, quantity, city, whether the supplied shop data could answer the question, and a customer-facing reply. The benchmark therefore measures more than fluency. It checks whether a model can parse compressed language and keep its answer grounded in the fictional shop’s information.
Scores reported for the eight models
The table reproduces the benchmark author’s reported scores for formal messages, slang, slang with typos, and a separate no-guard task. They are not independently reproduced results. The article says the Kaggle presentation includes 95% intervals and cautions that differences within those intervals may be noise; the available account does not establish that small score gaps are statistically meaningful.
| Model | Formal | Slang | Slang with typos | No-guard task |
|---|---|---|---|---|
| Claude Sonnet 5 | 99.8 | 99.4 | 99.4 | 100 |
| Gemini 3.7 Flash | 99.7 | 99.7 | 99.1 | 100 |
| GPT-5.5 | 100 | 98.9 | 99.1 | 99.4 |
| Gemini 3.1 Flash-Lite | 98.5 | 98.2 | 97.0 | 95.4 |
| Gemma 4 26B A4B | 98.1 | 98.1 | 95.7 | 98.1 |
| Claude Haiku 4.5 | 97.2 | 92.7 | 91.5 | 93.5 |
| gpt-oss-20b | 98.0 | 91.6 | 82.8 | 86.4 |
| GPT-5.4 nano | 92.7 | 89.2 | 78.1 | 90.7 |
The author reports that the top three models changed little between formal and typo-noisy input in this run. The reported typo-noisy scores fell more for GPT-5.4 nano and gpt-oss-20b. That is a useful signal about which examples challenged these runs, not proof of a general ranking for Indonesian or of how the models will behave on actual shop conversations.
What the no-guessing instruction changed
The author also ran a no-guard task on 27 difficult slang messages, removing the system instruction that told the model not to guess and to say an administrator would check. The reported trap-message failure rate for gpt-oss-20b rose from 11% to 19% without that instruction; Claude Haiku 4.5’s rose from 4% to 15%. The author reports zero trap failures for GPT-5.5, Sonnet 5, and Gemini 3.7 Flash both with and without the line.
This comparison suggests that an explicit abstention instruction can matter for some models in this test. It does not show that a prompt alone makes a customer-support system safe: the task was limited to 27 selected messages, and the benchmark does not establish behavior across longer conversations or real operational conditions.
The scoring was Python-based, not an LLM judge. The author describes half the score as understanding and half as grounding, with answerability and factual response content checked against the shop data. As the author puts it, “I wanted to be able to point at every lost point, so the scoring is plain Python.”
Where the results can mislead
A small, hand-written, single-turn test
All 84 base messages were written by the benchmark author, and the test is single-turn. Three fictional shops and manually composed examples can expose specific parsing and grounding problems, but they cannot represent the range of customers, products, regions, spelling habits, or follow-up exchanges that real Indonesian businesses encounter.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Grounding checks have blind spots
The author says unsupported numbers were penalized strictly, but a numeric-focused check can miss a fabricated claim that contains no number. One example involved an unsupported statement about fabric quality; in that case, the answerability flag caught the problem. The example illustrates why a correct-looking score is not a substitute for checking both the extracted fields and the customer-facing wording.
Best Value
Scores and costs are author-reported snapshots
The article reports estimated costs per 1,000 customer messages of about $12 for GPT-5.5, $6.50 for Claude Sonnet 5, $3 for Gemini 3.7 Flash, and $0.40 for Gemini 3.1 Flash-Lite. These are the author’s benchmark figures, not verified current provider prices or a generally reproducible bill; actual cost depends on usage and the pricing in effect when a system is run.
For the same run, the author says Gemma 4 26B A4B took about 25 seconds per reply and used around 1,200 thinking tokens. This is a reported observation from that setup, not a general latency guarantee.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What a shop or evaluator can take from it
UMKM-Bench is most useful as a reminder to test the language and information boundaries of a specific support workflow—not to pick a model from a leaderboard alone. For a shop building its own evaluation, the benchmark’s design points to several practical checks:
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minute- Include the actual shorthand, misspellings, and regional vocabulary customers use, and check ambiguous place names against the shop’s delivery rules.
- Test both answerable questions and questions whose answers are missing from the catalog or policy data.
- Check extracted details such as product, quantity, and city separately from the reply a customer sees.
- Include misleading claims, arithmetic, and instructions embedded in customer messages, then verify that the model follows the shop’s data rather than guessing.
- Compare behavior with and without explicit abstention instructions, while treating those instructions as one control rather than proof of production safety.
- Measure cost and response time in the intended setup; the figures in this article are not current price quotes.
The author suggests extending the test with multi-turn chats, voice notes, more regional forms such as Minang, Batak, and Makassar slang, and scoring that accounts for price. Those additions would address gaps in what a small single-turn test can establish.
Reproducibility and source access
The DEV article links a Kaggle benchmark and a GitHub repository and says a new shop can be added through data/stores.json plus messages. The linked Kaggle and GitHub pages could not be inspected for this account, so the dataset contents, code, licensing, current availability, and independent reproducibility are not established here. The reported scores should therefore be read as the author’s results from this particular benchmark setup.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
