Mayank Kaul’s experiment found that removing a 1,350-line CLAUDE.md changed compliance with two of 20 checkable rules—but both rules conflicted with patterns visible in the repository. In 30 runs across six configurations, the full-file setup passed 399 of 400 checks, according to Kaul. The result is a useful case study in how coding agents may follow repository evidence over written instructions, not proof that long instruction files are generally unnecessary.
What did the experiment test?
Kaul tested whether a coding agent would follow project rules with or without a large CLAUDE.md file while completing one ticket-sized task. The file contained 1,350 lines. He selected 20 mechanically checkable rules and used a deterministic checker to inspect the resulting file tree, rather than relying only on the agent’s own account of what it had done.
![Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results](https://m.media-amazon.com/images/I/415+fSJacsL._SL160_.jpg)
As an Amazon Associate I earn from qualifying purchases.
The experiment comprised 30 runs across six configurations. With the full file present, 399 of 400 rule checks passed. Kaul reports that two rules accounted for the difference associated with removing the file.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallWhich two rules changed the outcome?
Where to put the changelog entry
The instruction file said to add a bullet under ## Unreleased. But the repository’s changelog kept “Nothing yet.” under that heading and placed actual entries in released sections. Its header also described a workflow in which a new version section is cut. Runs without the instructions followed that visible structure instead of adding an unreleased bullet.
#1 Best Overall
Which version number to bump
The file instructed the agent to bump the patch digit. Repository history, however, showed a minor-version increase when a change added public API surface. That history gave the agent evidence for a different versioning decision than the written rule.
Kaul calls these outcomes rule violations, but they were also defensible choices based on the repository’s demonstrated conventions. The disagreement is the important detail: the file prescribed one behavior while the changelog and version history suggested another.
Why did the repository matter when the file was absent?
The experiment was not a clean test of an agent that had never encountered a project contract. Kaul says the initial setup had confounds: the deleted file remained recoverable from Git history, while memory mechanisms and a test-harness comment also pointed toward the rules file. He rebuilt the base repository so the file was absent from every scanned object-store blob.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Even after that change, he characterizes the file-absent runs as a model told that a contract existed but unable to access it. The repository itself still supplied clues through its current changelog and past version changes. Because the repository included several examples of those conventions, they were unusually easy to infer. A project with little history—or a newly created repository—might not offer the same evidence.
How did the configurations compare?
Kaul reports the following scores and per-run costs for two configurations. These are results from this experiment, not general estimates of model performance or price.
| Configuration | Rule-check result | Reported cost per run |
|---|---|---|
| Nineteen-line brief plus a Skill | 100/100 | $0.696 |
| Full 1,350-line file | 99/100 | $1.094 |
Kaul says the full-file configuration cost more in each of 25 pairwise comparisons. The small score difference and cost comparison apply only to the tested task, repository, model, and setup; they do not establish that a compact brief or Skill will outperform a long file elsewhere.
What can teams take from this case?
The practical question is not simply whether a project instruction is long or short. It is whether each rule adds guidance that cannot already be inferred from the codebase—and whether it is consistent with the evidence the repository presents.
- Check rules against the repository. For each instruction, ask whether a careful reader could infer the same behavior from current code, documentation, changelogs, or history.
- Investigate contradictions before editing the rule. If a rule conflicts with established practice, determine whether the rule is stale or whether the repository convention is undesirable. The right fix may be to update the instruction, change the project practice, or document why the written rule intentionally differs.
- Try a real task more than once. Observe whether a rule changes the resulting files, rather than judging it only by how sensible it sounds or by an agent’s verbal confirmation.
Kaul’s interpretation captures the tension: “The rules worth keeping are the ones your codebase contradicts, which is uncomfortable, because those are also the ones most likely to be wrong.” That is his reading of this experiment, not a general rule that repository conventions should always override explicit instructions.
Best Value
What the result does not establish
The study covered one task, one repository, one model, and five runs per arm. Kaul reports no statistical testing on the compliance outcomes. It therefore cannot establish that shorter instruction files generally perform as well as longer ones, that Skills are universally preferable, or that agents will infer conventions reliably in a greenfield repository. The finding is narrower: in this repository, two written rules disagreed with evidence in the project, and removing the file changed how those rules were followed.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Free tools Windows power users keep installed
One-click scans. No signup required.
