The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Treat a SKILL.md file the way you would treat configuration that changes production behavior. Define what the skill must make the agent do, test the prompts where it should and should not activate, capture each run, and grade the result against explicit checks. Repeat the same checks after every change. Testing does not guarantee reliability. What it does is make intended behavior measurable, so regressions show up instead of surfacing later as surprises.
What SKILL.md is, and where the analogy stops
A skill is a reusable workflow. Its SKILL.md file carries the metadata and instructions that describe what the workflow is for and how the agent should carry it out. Supporting resources can sit alongside the file in the same skill directory, according to OpenAI’s Skills documentation on the OpenAI API and its “Build skills” guidance for Codex.
As an Amazon Associate I earn from qualifying purchases.
“Production config” is an analogy, and it is worth being precise about its limits. SKILL.md is Markdown with metadata, not executable application configuration. Its effect runs through a model reading and applying the instructions, which is why the same file can produce different runs and why testing matters. OpenAI’s Skills documentation notes compatibility with the Agent Skills standard and validation of front matter. Validating the header catches malformed metadata. It cannot show whether the instructions produce the right behavior.
Why the description decides whether the skill runs at all
The description in the front matter shapes when the model considers invoking the skill. A skill can be well written and still never fire on the requests it was built for, or fire on requests it was never meant to handle. For that reason, a test set needs two kinds of prompts: intended requests that should activate the skill, and nearby requests that should not. The rest of this loop is built around that distinction.
Step 1: Write down what success means before you edit
Before you revise the skill, state four things: the outcome, the required steps or tool calls, the output conventions, and any limits the agent must honor. Keep the first version to must-pass criteria in four groups:
- Outcome: the requested task completes and the required artifacts exist.
- Process: the expected steps and commands occur.
- Style: formatting and project conventions match the stated requirements.
- Efficiency: the run avoids unnecessary commands and excessive token use while still meeting requirements.
Do not try to encode every preference at once. A short must-pass list is easier to grade consistently, and it stays stable enough to compare across versions.
Step 2: Build a small, varied prompt set
OpenAI’s Codex guide on testing agent skills, written by Dominik Kundel and Gabriel Chua and published January 22, 2026, suggests starting with 10 to 20 prompts for a single skill, then adding prompts as real misses appear. That range is practical early-stage guidance from OpenAI. It is not a universal minimum and it is not a statistically derived sample size. It is a workable scale for catching regressions and confirming early improvements, not a measure of complete coverage.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →The prompts should vary in type. The table below uses an illustrative skill that writes changelog entries from a git diff. The examples are invented for this walkthrough, not taken from OpenAI’s guide.
Rank #3
| Prompt type | Example for the changelog skill | Expected behavior |
|---|---|---|
| Direct invocation | “Use the changelog skill to write release notes for this diff.” | Skill activates and produces an entry |
| Indirect, on-target | “Summarize what changed in this branch for customers.” | Skill activates and produces an entry |
| Realistic contextual | A pasted pull request thread plus a request to draft the release notes | Skill activates; entry reflects only the changes in the thread |
| Negative control | “Explain what a changelog is and why teams keep one.” | Skill does not activate |
| Adjacent request | “Review this diff for bugs.” | Skill does not activate |
| Incomplete input | “Write the release notes,” with no diff attached | Skill flags the missing input; no changes are invented |
Negative controls and adjacent requests deserve as much attention as the direct prompts. Without them, a skill that fires on everything will look like it is working.
Step 3: Capture each run as an eval record
OpenAI describes an eval as a prompt, a captured run with its trace and artifacts, a set of checks, and a score that can be compared over time. The Codex guide puts the purpose plainly: “Evals (short for evaluations) check whether a model’s output, and the steps it took to produce it, match what you intended.”
For each prompt in the set, record:
- The exact prompt text and the skill version under test.
- Whether the skill activated.
- The sequence of actions the agent took, including commands run.
- The artifacts the run produced, such as files or the final output.
- The result of each check and the overall score.
Capturing the trace matters because a correct final output can hide a wrong process. If a changelog entry is accurate but the agent skipped the step that reads the diff, the outcome passes and the process check fails. You want both to show up.
Step 4: Grade observable requirements deterministically, and judgment with a rubric
OpenAI’s guide recommends combining two grading methods. Use deterministic checks for anything you can observe as a yes or no: required files exist, a named command ran, the output contains required fields, a forbidden phrase is absent. Use a rubric for qualities that resist a simple assertion, such as whether the tone fits the project or whether a summary faithfully reflects the diff.
Best Value
- Deterministic example: the output file exists, the entry has a version heading, and no commit hash appears in the customer-facing text.
- Rubric example: each bullet describes one user-visible change, uses the project’s tone, and makes no claim that the diff does not support, scored on a fixed scale.
Store the rubric text next to each score. A score without its standard cannot be compared with a score from a later run, and that comparison is the point of the exercise.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Step 5: Separate misses from false positives
A skill that fails to activate and a skill that activates when it should not are different problems, and output quality is a third, separate question. Inspect them in that order.
| Symptom in the run record | What it indicates | Where to look first |
|---|---|---|
| Skill does not activate on an intended prompt | Discovery problem: the description may not signal the triggers users actually write | Description wording, compared with prompts that did activate |
| Skill activates on an adjacent or negative prompt | Trigger-boundary problem: the description is broader than intended | Description scope and the negative-control results |
| Skill activates correctly but fails output checks | Instruction or output-quality problem | Process steps and output conventions in the instructions |
| Output states facts absent from the input on an incomplete prompt | Robustness problem | Instructions covering missing inputs and unsupported assumptions |
Fixing the wrong layer wastes time. Rewriting the instructions will not repair a description that never matches the request in the first place.
Free tools Windows power users keep installed
One-click scans. No signup required.
Step 6: Turn real failures into regression cases and compare versions
Every real failure becomes a new prompt in the set. Keep the set as a living record, and rerun all of it after each change using the same must-pass checks. A fix for one prompt can quietly break another, and a full rerun is the only way to see that. When you compare two versions or two approaches, run both against the identical prompt set and compare along the same axes:
Quick Recap
| Axis | What to measure | Evidence of a pass |
|---|---|---|
| Trigger precision | Intended direct and indirect prompts activate; adjacent requests do not | Activation recorded per prompt in the run |
| Outcome correctness | The requested task completes and required artifacts exist | Deterministic checks on files and outputs |
| Process adherence | Expected steps and commands occur | Action sequence in the captured trace |
| Output quality | Formatting and project conventions match the stated requirements | Rubric score, recorded with its rubric text |
| Efficiency | No unnecessary commands and no excessive token use | Command count and token use in the run record |
| Robustness | Incomplete inputs and edge cases produce no invented facts or unsupported actions | Edge-case checks in the prompt set |
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
