What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
DarijaBench suggests that several AI models can handle a small set of written Moroccan Darija tasks, but it does not prove broad or reliable understanding. Its creator, Soufian Zaari, reports high scores for four models on 60 items; the reported score gap is not statistically significant, and the test does not establish performance on speech, regional variants, or everyday code-switching.
What did DarijaBench test?
Zaari describes DarijaBench as a 60-item benchmark with 20 items in each of three categories: translating Darija into French, classifying sentiment, and answering Moroccan cultural questions written in Darija. The overall score averages results across those categories. Answers are graded automatically with keyword checks. Source: Soufian Zaari’s DarijaBench report.
As an Amazon Associate I earn from qualifying purchases.
That makes the result a snapshot of performance on these particular prompts and task types—not a general measure of language understanding. Keyword grading can also mark a correct answer wrong when it uses different wording from the expected keywords, a limitation Zaari acknowledges.
Free tools Windows power users keep installed
One-click scans. No signup required.
What scores did the models receive?
| Model | Reported overall score |
|---|---|
| Gemini 3.7 Flash | 0.95 |
| Claude Sonnet 5 | 0.95 |
| Claude Opus 4.7 | 0.95 |
| GPT-5.6 Luna | 0.90 |
These figures are Zaari’s reported results for the 60-item benchmark, not independently replicated scores. In an October 2, 2026 update, he reported an exact McNemar p-value of 0.25 for the comparison and described the outcome as a four-way tie at n = 60. The apparent five-point difference therefore does not establish a winner.
#1 Best Overall
Which tasks were easier or harder?
Sentiment classification
Sentiment was the easiest category in Zaari’s report. Gemini 3.7 Flash scored 20 out of 20 on its sentiment items. That result describes this category in this benchmark; it should not be read as a general accuracy rate for Darija sentiment analysis.
Darija-to-French translation
Translation was the hardest category, with Gemini 3.7 Flash reported at 85%. Zaari points to idioms and French loanwords whose usage varies in Darija as sources of difficulty. The result is limited to the benchmark’s 20 translation items.
Moroccan cultural questions
The third category asks cultural questions in Darija. The report includes this task in the overall score, but a set of 20 questions cannot establish how well a model handles the full range of Moroccan cultural knowledge or context.
What does the small sample size mean?
Sixty items provide limited evidence for distinguishing close scores. Zaari says that reliably detecting a five-point gap would require roughly 200–300 items. That is the benchmark author’s estimate, not an independently verified power analysis. A larger test would still need representative prompts and a grading method that accommodates valid alternative answers.
Other important dimensions remain outside what this benchmark establishes: spoken Darija, regional variation, mixed-language conversation, and variation in writing scripts. Zaari identifies idioms, code-switching, and regional variants as areas for a harder future version.
How does this benchmark relate to other Darija evaluations?
Darija-specific adaptation has shown promise in a separate evaluation, but it is not a replication of Zaari’s result. The Association for Computational Linguistics record for Shang and colleagues’ Atlas-Chat paper, published in January 2025, describes fine-tuned models in 2B, 9B, and 27B sizes. It reports that Atlas-Chat-9B achieved a 13% performance boost over a larger 13B model on DarijaMMLU. That finding concerns another model and another evaluation suite; it cannot confirm the scores above. ACL Anthology record for the Atlas-Chat paper.
A separate AtlasIA project describes a community arena using nearly 300 Darija prompts, head-to-head outputs, and user votes for accuracy, fluency, and cultural alignment. Human preference voting is a different evaluation method from keyword-based automatic grading. The project post discusses older model versions, so it should not be treated as a current leaderboard without checking the live project. AtlasIA project.
There is also a different resource with the same name: the MBZUAI-Paris DarijaBench dataset card. It describes a broader research dataset for summarization, six translation directions, and sentiment analysis. It is not the 60-item benchmark discussed in Zaari’s report. MBZUAI-Paris DarijaBench dataset card.
Best Value
How should you judge claims about Darija AI?
When comparing model results, check whether the evaluations use the same tasks, sample size, scoring method, script and code-switching patterns, regional coverage, and model versions. Automatic keyword checks and human preference votes measure different things, and results from separate suites should not be combined into a single ranking.
For Zaari’s four-model result specifically, the available report is evidence about a compact set of written prompts, not proof of broad understanding. No independent replication of this specific comparison is established in the cited sources, and its results do not resolve how models perform on speech or everyday regional and mixed-language communication.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
