A required CI gate, realistic end-to-end tests, human review, and production monitoring helped Geoff Cox manage AI-generated changes in TopSet, a live-trading system he says runs on his own capital. But his account is not an independent audit, and a green build did not prove the code was correct: tests passed while false fixtures, NaN scores, and an unexecuted code path hid defects. The useful lesson is not that a test suite makes trading code safe; it is how to build several checks that can challenge one another.
In a 2025 retrospective on DEV Community, engineer Geoff Cox described using Claude Code, Cursor, and Copilot while building TopSet. He reports about 1,600 pull requests over 21 months, with a roughly ten-minute required pull-request suite and a broader weekly test suite. He also says the project was a one-person effort using his own capital, a context that does not establish what would be sufficient for a larger team or another trading system.
As an Amazon Associate I earn from qualifying purchases.
What did the pre-merge gate check?
Cox says every pull request had to pass checks spanning code quality, application behavior, database integrity, migrations, and infrastructure. A failure blocked the merge.
| Check | What Cox says ran on each pull request |
|---|---|
| Code quality | Ruff linting and mypy type checks. |
| Automated tests | About 7,500 tests across 331 test modules. |
| Database state | A check that the test database was empty afterward. |
| Migrations | Rollback and reapplication of migrations from the base state. |
| Infrastructure | Terraform validation against two AWS accounts. |
The layers target different kinds of mistakes: style and type checks catch some defects before execution; tests exercise behavior; database checks can expose leaked state; migration checks challenge whether schema changes can be reversed and replayed; infrastructure validation checks configuration. None independently establishes that a strategy, order, or calculation is financially correct.
#1 Best Overall
Why add a weekly suite if pull requests are already green?
The per-change gate was quick enough to run before merge, while a scheduled suite exercised slower or less predictable behavior. Cox says the weekly suite included two full rebalancing end-to-end tests against a mock broker, one integration test repeated 100 times, nine deterministic model-training snapshots pinned in Docker, and two training-pipeline leakage checks.
The mock broker varied fill timing, partial completion, and prices. Repeating an integration test under changing interleavings can expose timing-sensitive failures that a single deterministic run misses. The point is not that 100 runs prove the absence of races, but that a test is more likely to encounter a problematic ordering when the workflow is deliberately varied.
Trading workflows are especially sensitive to persisted state and interactions between orders, fills, and account events. Cox’s examples included:
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →- Resubmitting orders.
- Resuming a rebalance with partial fills, or with buys and sells still outstanding.
- Cancelling smart orders while they were in flight.
- Handling deposits or repeated withdrawals that arrived during a rebalance.
These cases test more than whether one function returns an expected value. They challenge whether the system can resume consistently when only part of a multi-step financial operation has completed.
How did green tests conceal real defects?
Cox’s examples show several ways a test can agree with faulty code: its fixture can encode the same false assumption, invalid inputs can make two outputs look equal, or the changed logic may never run. These are failures of evidence, not merely failures to write enough assertions.
A fixture can preserve a false data convention
Cox says his code treated a broker’s stock-dividend rate as a fraction such as 0.05. The records he examined instead represented new shares divided by old shares: all 101 records across 43 symbols had rates of at least 1.0016. A test fixture repeated the mistaken convention, so implementation and test agreed while both were wrong.
He reports that one nine-event price history was deflated by about 490×, producing an apparent rise of more than 1,000-fold. Those figures describe Cox’s examined data and example, not stock-dividend records generally. His experience illustrates why fixtures should be checked against source records rather than treated as authoritative just because tests pass.
Recommended Free Tools
NaN can make agreement meaningless
An alternate model selector appeared to agree 100% with the existing selector. Cox found that missing input columns made all the scores NaN, and both selectors then fell through to the same fixed tie-break. Agreement therefore said nothing useful about whether the alternate scoring logic worked.
He says the fix made missing or invalid scoring inputs a hard error and pinned the test to a version containing the required columns. In a numerical pipeline, checking that inputs are valid is part of checking the output: two algorithms can return the same result for the wrong reason.
A changed field may never reach the changed code
A new training target initially produced the same picks as the baseline. Cox reports that all 238 of 238 picks were identical because the runner routed to a different function that ignored the new field. After correcting the route, zero of 238 picks were identical, and a regression test required the modes to differ.
As Cox put it, “Identical results are not a pass. They’re a smell.” Identical output can be legitimate, but when a change is expected to affect behavior, the team needs to establish that the relevant path ran and received the intended inputs. A test that merely records output without checking those conditions can certify a no-op.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallA zero-value order can trigger retries through rollback
Cox also describes a pending buy adjusted for withdrawals reaching zero. A guard that rejected only negative values let zero through, and an affordability condition accepted 0 ≥ 0. The database then rejected the order because of an amount-greater-than-zero constraint. Transaction rollback discarded the completing order’s status update as well, leaving the scheduler to retry with unchanged inputs.
The first proposed correction guarded promotion of the order, but missed mutation of a live ORM object that could be persisted by autoflush. Cox says he reproduced the failure and reran the proposed fix against that reproduction. The sequence matters: a database constraint can catch an invalid state, but transaction boundaries and in-memory object changes determine whether recovery actually makes progress.
How can a team check that a change did what it was meant to do?
Tests are useful evidence about behavior specified by their assertions; they are not an independent truth source. Cox’s line, “A test asserts that a function returns what the author believed it should return. A spreadsheet asks whether the number is true,” captures the need to verify calculations outside the code path that produced them.
- Validate sample data against raw source records, especially when a field’s meaning or units are easy to misread.
- Make absent columns, invalid values, and impossible states fail explicitly instead of allowing a plausible fallback.
- For a change expected to alter behavior, verify both that the relevant route executed and that its inputs contained the new field or setting.
- Inspect exported data and arithmetic independently, then spot-check the underlying inputs.
- Investigate output that is unexpectedly identical, unchanged, or suspiciously perfect; agreement is useful only when the compared computations received valid inputs and exercised the intended logic.
This is why a green build should be read as “the configured checks passed,” not “the financial behavior is correct.” As Cox also wrote, “Tests prove that the code does what the tests say.”
Free tools Windows power users keep installed
One-click scans. No signup required.
What still requires human ownership?
Automation can enforce a chosen definition of done, but people decide whether that definition covers the actual risks. Cox says he reviewed architecture, test design, and merge decisions himself, with deeper attention to model-training and execution paths and reliance on tests for line-level behavior.
Best Value
That division is not a reason to skim agent-written changes. The reviewer must decide whether a test’s assumptions reflect production data, whether edge cases match the workflow, and whether the outcome is plausible—not merely whether the diff looks tidy. Cox’s principle is direct: “The author owns the change, whoever typed it.”
He advises teams to strengthen the gate before increasing agent throughput and to judge the process by escaped defects and hotfix rate rather than lines of code or pull-request counts. These are recommendations from his experience, not a controlled measurement of AI-agent productivity or defect reduction.
Why monitor after deployment?
Pre-merge checks cannot anticipate every production condition. Cox says CloudWatch production ERROR alarms were sent to Discord; he used logs and a local database copy to investigate incidents. He explicitly notes that tests did not catch the first failures described in his account. Monitoring helped surface them, and the fixes and regression checks helped prevent recurrence.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesThat makes production monitoring a separate safety layer rather than an extension of the green build. A live system needs a way to notice errors after deployment, inspect relevant state, and determine whether a failure is repeating. A test gate reduces some risks before merge; it cannot establish that the system is behaving correctly right now.
What the account does—and does not—establish
Cox’s account is a practitioner’s retrospective, not an independent audit or controlled study. It supports a practical case for combining required checks, realistic workflow tests, independent validation, human review, and monitoring. It does not show that this particular suite guarantees safe trading, that the reported results generalize, or that a one-person project’s safeguards are sufficient for a larger engineering organization.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
