You can independently reproduce Sthanika’s released public Jev Decision Index run, but not its headline Full score. The public-kit procedure yields the authors’ comparable public-index result when you use the same model revision, Decision Index 0.3 kit commit, software versions, temperature, answer-slot protocol and hardware class. The Full score combines public and privately administered tests, so its exact value cannot be recreated from the released artifacts alone.
Know which score you are trying to reproduce
Sieve-31B has two scores associated with Decision Index edition 0.3.2. The model card lists a Full score of 63.8, based on board data dated 14 October 2026. It also lists the authors’ run on the released public suite: a public index of 62.52, with a raw score of 71.08 and breadth of 61.55.
As an Amazon Associate I earn from qualifying purchases.
| Measurement | What it contains | Can an outside researcher reproduce it? |
|---|---|---|
| Public index | The released Decision Index suite | Yes, using the official kit, engine and matching settings |
| Full score | 20% public benchmarks, 50% private tests of the same skills and 30% private tasks from new domains | No. The private data is controlled by the benchmark maintainers |
Therefore, a successful public-kit rerun is evidence of parity with the public evaluation, not an independent recreation of 63.8. Treat those as separate measurements whenever you publish or compare results.
What you need before running the benchmark
Use the card’s software versions
Create a clean Python environment and install the versions specified for Sieve-31B:
#1 Best Overall
- [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
- [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
- [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
- [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
- [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.
pip install transformers==5.17.0 peft==0.21.0 accelerate==1.15.0 huggingface_hub==1.32.0 jinja2 httpx pydanticDo not silently substitute a newer Transformers, PEFT or Hugging Face Hub release. A change in tokenizer behavior, adapter loading or generation defaults can alter scores even when the model files are unchanged. Record your Python version, CUDA/runtime stack and the exact installed package versions alongside the benchmark output.
Download the adapter and its exact base
Fetch sthanika-ai/Sieve-31B and the base model google/gemma-4-31B-it at revision 842da37. Sieve-31B is a rank-128 LoRA adapter with 979 million trainable parameters; the Gemma 4 31B Instruct base remains frozen and is loaded in bfloat16. Loading a different base revision, or merging the adapter into a modified checkpoint, is not the same evaluation target.
Make sure the account running the job can read both repositories and that the local cache has enough space for the base weights, adapter files and generated run artifacts. Keep the revision identifier in your experiment log so another person can verify what was evaluated.
Check out the official Decision Index kit
Use Decision Index 0.3 and check out commit 62d2f51. The commit is a reproducibility anchor: a moving branch can contain changed prompts, scoring code or task metadata. If you already have a clone, switch to that commit before invoking the pipeline and record the resulting commit in your report.
Rank #2
- 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
- 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
- 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
- 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
- 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.
Run the public evaluation
- Activate the clean environment. Confirm that the pinned packages import successfully and that the process can see the intended GPU or GPUs.
- Place the model and kit where the engine can resolve them. The model option below refers to the Hugging Face adapter identifier; the engine must also be able to resolve its Gemma base.
- Launch the official engine. From the Decision Index kit checkout, run:
python -m decision_index pipeline
--engine sieve.decision_index_engine:SieveEngine
--option model=sthanika-ai/Sieve-31B
--out runs/Sieve-31B- Preserve the output directory. Keep the request-level results, logs and configuration under
runs/Sieve-31B. They are needed to distinguish a scoring disagreement from an inference failure. - Compare only after completion. A partial run, an interrupted worker or silently skipped request cannot establish parity with the published public result.
Memory, batching and request limits
The card specifies approximately 83 GB of bfloat16 memory for this evaluation. Plan for one GPU with 96 GB or two GPUs with split loading. A smaller device may fail during model loading or force settings that no longer match the reference run.
- Keep every prompt row at or below 32,768 tokens.
- Keep each question within the benchmark’s 255-option limit.
- Use bfloat16 for the frozen base as specified; changing precision can change logits and therefore choices.
- Document whether weights were split across devices, plus GPU model, memory and any offload configuration.
These limits are part of the evaluation contract, not suggestions for truncating oversized inputs. If a row exceeds the context limit, fix the input or task packaging rather than silently truncating it.
Match Sieve-31B’s decision-output interface
Sieve-31B is evaluated through typed decision outputs rather than unconstrained prose generation. The engine’s answer-slot interface supports choice, noul and score question types. Use the engine’s native rendering and parsing instead of writing a custom prompt that asks the model to explain its answer.
Recommended Free Tools
The model card specifies a fixed temperature of 8.5495. Keep that value unchanged for a parity run. A lower “more deterministic” temperature, a different sampling implementation or post-processing that converts free-form text into options can produce a different result while still appearing to run the same tasks.
Rank #3
- [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
- [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
- [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
- [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
- [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by
For each request, retain the typed answer, validity status and any parser or transport error. This lets you determine whether a score changed because the model selected a different option or because a request never produced a scoreable answer.
Validate the run before comparing headline scores
The authors report 140,178 of 140,178 scoreable requests completed successfully: zero unsupported requests, zero errors and no truncation. Use that completion profile as your first parity check. A numerically similar index with skipped or failed requests is not equivalent.
| Public area | Reference score |
|---|---|
| Knowledge & Reasoning | 49.9 |
| Language Understanding | 65.6 |
| Retrieval & Classification | 69.5 |
| Tools & Automation | 78.7 |
| Arts & Human Taste | 43.5 |
Check the five area values and request counts before looking at the aggregate public index. If one area is absent, unusually low or based on fewer requests, investigate the task logs first. The aggregate should be the final comparison, not the only one.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Why your result may be 62.52 instead of 63.8
You ran the public suite, not the Full board
This is the expected explanation. The released kit measures the public component. The Full score adds privately administered tests and new-domain tasks using the weighting shown above. Because those private prompts and labels are not released, no command against the public kit can recreate the complete board calculation.
Rank #4
- Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
- Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
- All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
- Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
- Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.
The kit or model revision differs
Verify Decision Index commit 62d2f51, adapter/base revision 842da37, and the exact package versions. A branch tip, a re-uploaded model snapshot or a different Gemma checkpoint can alter prompts, tokenization or adapter application.
Requests were dropped, truncated or marked unsupported
Compare your completed-request count with 140,178. Inspect logs for unsupported question types, transport errors, parser failures and context truncation. Correct those failures before interpreting any score difference as model behavior.
Inference settings changed
Confirm bfloat16 loading, the fixed temperature of 8.5495, the answer-slot interface and the 32,768-token and 255-option limits. Device splitting is acceptable within the stated memory class, but changing precision, sampling or output parsing is a methodological change.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
You compared different editions or score definitions
Write the edition, kit commit and score definition next to every number. “Decision Index score” is ambiguous unless you state whether it is the public index or the Full score and identify the board edition used.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to report a reproducible result
A useful report makes another evaluator able to distinguish a true model difference from an environmental difference. Include:
- Model adapter name and revision, plus the frozen base name and revision.
- Decision Index edition and kit commit.
- Transformers, PEFT, Accelerate and Hugging Face Hub versions, along with the Python and CUDA/runtime versions.
- GPU model, memory, number of devices, precision and any split-loading or offload settings.
- Temperature, answer-slot protocol and the maximum prompt and option limits enforced.
- Total scoreable requests, completed requests, unsupported requests, errors and truncations.
- The five public area scores and the aggregate public index.
- The output directory or an archived manifest containing configuration and request-level status.
When comparing another implementation or model, use the same axes: public versus Full definition, benchmark kit and commit, model/backbone revision, request coverage and error count, area scores, temperature and output protocol, and hardware or latency conditions. Reporting only one headline number hides the variables most likely to explain a disagreement.
What “exact reproduction” means in this case
Exact independent reproduction is available for the released public run: the artifacts, engine command and settings are specified well enough to rerun that measurement. Exact independent reproduction of the Full score is explicitly unavailable because 80% of its weighting comes from private components (50% private tests of existing skills and 30% private tasks from new domains). Only the maintainers can execute those hidden requests and apply their labels.
Bharat Lingam, CEO and co-founder of PurpleTalk, described the release goal this way: “We are publishing everything needed to reproduce our results, and we invite the research community to test it, challenge it and build on it.” In practice, that invitation applies to the public evaluation artifacts; a public rerun should be presented as public-index validation rather than as a replacement for the private-board result.
Quick Recap
Quick parity checklist
- Decision Index 0.3, commit
62d2f51. sthanika-ai/Sieve-31Bwithgoogle/gemma-4-31B-itat revision842da37.- Transformers 5.17.0, PEFT 0.21.0, Accelerate 1.15.0 and Hugging Face Hub 1.32.0.
- Approximately 83 GB bfloat16 memory: one 96 GB GPU or two split-loaded GPUs.
- Fixed temperature 8.5495 and the native
choice,noulandscoreanswer slots. - No prompt row over 32,768 tokens, no question over 255 options.
- 140,178 scoreable requests completed with zero unsupported requests, errors or truncation.
- Area scores checked before comparing the aggregate public index.
- Any result labeled with its score definition, edition, revisions and request coverage.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
