Case studies
Four short engineering stories from these repositories. None of them is hypothetical: every one happened while building the projects on this site, and every claim below links to the code, test, or document that backs it up, so you can check the story against the source rather than take my word for it.
dc-twin ships a small browser explorer next to its API: plain JavaScript, one stylesheet, served by the same FastAPI process that runs the simulator. It worked on Linux CI, it worked in the Docker image, and the test suite was green. Then I started the server on a Windows machine to take a screenshot for the README, opened the page, and got the header and nothing else. No error dialog, no broken layout — a page that simply never came to life.
The browser console had the real story: it had refused to execute /app.js because the file arrived as text/plain. Python's mimetypes module builds its type table per platform, and on Windows it reads the registry; on that machine something had registered .js as text/plain, and the static-file handler passes along whatever mimetypes guesses. On its own a wrong type would still have run, because browsers historically sniff content. But every dc-twin response carries X-Content-Type-Options: nosniff — a header I had added deliberately so responses are interpreted exactly as declared. A correct security header plus a wrong declaration meant the browser did precisely what it was told: it treated the application's own script as plain text and discarded it, silently.
The fix is three lines next to the static mount in api.py, pinning the media types the web assets actually use — text/javascript, text/css, image/svg+xml — with a comment explaining why they are there. Because a fix that lives only in one machine's memory is worth little, a regression test in test_api.py now requests /app.js and /styles.css and asserts the served content types, so the combination can never quietly come back.
What stays with me is that every layer behaved correctly. The registry is allowed to define local types, mimetypes documents that it reads it, and nosniff did exactly its job. The failure lived in the deployment environment, not in any line of code, which is why reading the code could never have found it. The lesson: verify the deployed thing, not the code. The screenshot session was doing exactly that — by accident.
The fix · The regression test · The explorer, running
Hand arithmetic as an acceptance oracle
A simulator has an awkward property: its output looks equally plausible whether or not it is right. Tests that compare the engine against values the engine itself produced only prove consistency, not correctness. For dc-twin I wanted an oracle that owed nothing to the implementation, so the reference fixtures were designed backwards: choose numbers round enough that a reviewer can redo the entire calculation on paper.
Take the N topology — a single 1 MW path with no redundancy and a UPS holding 50 kWh of usable energy, which is 180,000,000,000 mJ. In its reference scenario the utility fails at 120,000 ms of a 600,000 ms horizon. Discharging at 1 MW, the UPS lasts 180,000,000,000 mJ ÷ 1,000,000 W = 180,000 ms, so it reaches zero at 300,000 ms. The utility returns at 420,000 ms, giving one interruption of 120,000 ms and 120,000,000,000 mJ unserved out of 600,000,000,000 mJ demanded. The service ratio is 480 ÷ 600 = 0.8: exactly 800,000 ppm, no rounding involved. Every division in that chain comes out even because the inputs were chosen so it would.
The arithmetic is written out in full in the reference topologies document and the 2N reference scenario. The expected values are committed as a JSON fixture that lives deliberately apart from the engine, and on every push a reconciliation script replays each scenario in CI and requires the normative integers to match exactly. If the engine ever drifts from the paper answer, the build fails, and the paper answer wins the argument until someone can show where the paper is wrong.
This is an old instinct from electrical work, where you sanity-check a tool against a hand calculation before trusting it with anything that matters. The lesson: when the answer is checkable by hand, make the hand answer the test — not a value the code once produced and somebody blessed.
The hand calculations · The committed expectations · The CI reconciliation
Keeping look-ahead out of the backtest interface
The classic way a backtest lies is look-ahead: a strategy that, somewhere in its code path, touches data from after the decision it is simulating. The usual defence is discipline — a convention that policies only read the past, enforced by review. The quant risk engine puts the guarantee where it can be tested instead: the policy interface exposes only observations strictly preceding the decision date, so the classic mistake cannot be written through the interface itself. Caller-supplied policies remain responsible for not accessing future data through external state or external data sources.
At every scheduled rebalance the engine hands the policy a PolicyWindow built by slicing the aligned series at the decision date. The object physically contains the dates and prices strictly before that date and nothing else; its constructor rejects any history that touches its own decision date. There is no flag to pass and no method to call that reaches later data, because the later data is not in the object. The tests assert, for every rebalance of a real run, that the window ends exactly one trading day before the decision.
A guarantee like that deserves evidence that it guards something real, so the test suite includes a deliberately cheating policy that reaches around the interface: it is handed the full price series on the side, through external state the window never sees. The fixture is mean-reverting by construction — the trailing winner is always the forward loser — which makes honest momentum the worst possible policy and a peeking one the best. Starting from 1000, honest momentum finishes near 810 while the cheat finishes near 1210, and the test pins both values. A companion test then shows that the cheat's first move — locating the decision date in its data — raises an exception when attempted through the window: the date is absent from the index, and the positions at or after it are out of range.
So the guarantee is stated, tested, and priced: the window exposes only prior observations, the tests verify that at every rebalance, and the cheating policy documents exactly how much reaching around the interface would be worth — evidence of what the interface protects against, not proof that no caller could ever cheat through external state. The lesson: put the guarantee in the interface where a test can hold it, then keep one working demonstration of why it matters.
The window · The cheating policy · The methodology notes
When the year doesn't fit
The roadmap for the PV-BESS dispatch model asked for a benchmark of a complete 8,760-hour scenario — a full year at hourly resolution — publishing the hardware, runtime, solver gap, and input hash. Measuring it produced a table that behaves until it doesn't: a day solves in 0.04 s, a month in 0.58 s, a quarter in 2.75 s, a half year in 22.5 s. The full year completed once, in 191 s.
Then I ran the identical problem again, and the second phase of the solve — a refinement pass that re-optimizes throughput at the optimal profit — found no feasible incumbent within a 1,200-second per-phase limit and was aborted. An earlier draft of the price series had already shown the same behaviour, giving no incumbent within 280 s. The pattern was consistent: the economic phase scales roughly linearly and took 13.7 s at a year, but the refinement phase dominates beyond roughly 2,000 intervals, grows super-linearly, and its runtime swings unpredictably with the exact price series and solver environment. The honest summary of the full-year row is that the same inputs may take three minutes or may not finish.
That left a choice about what to commit. A full-year sample that fails its own documented run would be worse than none, so the committed large fixture is the 744-hour month, which solves in about a second, and the benchmarks page publishes the full-year measurements anyway — including the aborted rerun — together with the environment, the input hashes, and the script that regenerates the exact year so anyone can repeat the attempt. The page states plainly that the reproduction command may or may not finish, and that this is precisely the published finding. The practical horizon on modest hardware is about a quarter; a rolling-horizon decomposition that would bound the refinement is recorded as future work, not claimed.
The lesson: when a stated goal meets a measured limit, publish the limit. A benchmark page that only shows the runs that finished isn't a benchmark — it's advertising.
The benchmark page · The committed fixture · The year generator