Forty-Six PDFs and the Fix That Made Things Worse First

You have forty-six PDFs for a literature review. You drop them into a single upload and wait. It should take a few seconds a file. Internally, it took forty. Multiply that out and you get an actual number from our own bug tracker: about thirty minutes for the batch, five of the forty-six lost outright. One came back "API error: 500." Another, Acute and Residual Effects of Marijuana in Humans. pdf, timed out. The other forty-one finished eventually, after waits long enough to open a second tab and forget why you came.

That is the moment this piece is about. The single error is almost beside the point. Trust in a tool gets decided by the thirty minutes bracketing it, the crawl before and the crawl after. People forgive one weird error. They abandon a tool over the crawl.

Internally this started life as Bug Report 022, filed the day someone noticed the crawl, and renamed three days later once it was obviously bigger than one bug.

Blame the easy target first

Our first instinct, like most people's, was to suspect the expensive outside stuff, the AI doing the actual reading, the servers hosting everything. We tested both directly. Just over a second per call, each way. Fast. The slowness lived entirely in our own code, which is the least comfortable place to find it, because you can't file a support ticket against yourself.

The response internally was blunt: "Stop looking for explanations that are outside the code that YOU wrote... the chances of a global-scale service being the source of the problem is about 1 in one trillion in my eyes. You are the weakest link." That framing turned out to be exactly right, and mildly humiliating.

The actual cause, once we looked inward, was almost boring. One process handled everything, every request the whole platform served, one thing at a time. Reading each document blocked everything behind it, all sharing one thread of attention. Forty-six people politely waiting behind a single door.

The prescribed fix for exactly this shape of problem is the one everyone reaches for first: take the parts that block and hand them to a pool of background workers, so the main thread stays free. We rated it "likely the single biggest win." It was the obvious call. It was also wrong.

The retest that finished nothing

We shipped it and reran the same batch that evening. Three minutes in: zero files completed. Zero. We killed the upload rather than find out what ten minutes of zero looked like.

Handing the blocking work to a pool of background workers had moved the bottleneck somewhere worse. Every document, once handed off, still had a tail of small follow-up saves to make, and every one of those saves had to line up for the same small crew of background workers, a crew built for maybe five to twelve jobs at a time, now facing roughly eighteen hundred requests from forty-six documents all firing at once.

Before the fix, forty-six files inched forward together, slow, roughly twelve minutes total, but every file actually finished. After the fix, forty-six files all reached for the same dozen hands at once, and nothing finished, because nothing could get its turn twice.

Slow and steady had become an actual standstill. We had fixed the wrong shape of the problem, at real cost, in front of a batch someone was actually waiting on.

The fix nobody wanted to be the answer

The real fix was a limit. A bigger server was on the table, and tempting, because it's purchasable. You throw money or config at a problem and feel like you did something. The report on this names the tempting options one by one and rules them out on purpose: a bigger platform. None of it.

The fix that worked was a hard cap on how many documents get processed at the exact same moment, tuned to the real number the system underneath could actually absorb without anything lining up behind anything else. The cap was four documents at a time, down from forty-six. It sounds like doing less. It is doing exactly as much as the system can carry, which turns out to be the entire difference between "finishes eventually" and "never finishes at all." More lanes don't help if the counter behind them only has four people working it. The fix is admitting how many people are actually working the counter, and building the line to match.

That one change, a ceiling, is what took the batch from thirty minutes with failures down to two or three minutes with none.

The push that ate thirty-nine files

We learned a second lesson the hard way, a day later. A batch of forty-six PDFs was mid-flight when a docs-only commit, no code change, just a correction to an internal doc, got pushed to the main branch. That triggered an automatic rebuild of the server. Twenty seconds after the batch started, the ground shifted under it. Roughly two minutes later, the old server was gone and a new one had taken its place.

Seven documents had finished cleanly before the swap. The other thirty-nine were mid-processing when their server vanished, and because the pipeline forgets all its work between restarts, all thirty-nine sat marked "processing" forever, waiting on a process that no longer existed.

Nobody wrote a bug in that story. The system forgets what it was doing the moment the ground moves, and pushing to production while a real batch is running is exactly how you find that out. We wrote ourselves a rule after: no pushes during testing, however harmless the change looks. It's a small, slightly embarrassing thing to need a written reminder for. It's also exactly the kind of rule you only write after watching thirty-nine files disappear on your own instructions.

What you get from all of this

None of this is visible from the upload screen, which is the point. What you get is a batch of documents that behaves the way you assumed it always did: forty-six files in, a couple of minutes later, forty-six files out, none silently lost to a background process that ran out of hands. What used to sit at critical severity is, on paper, a routine Tuesday now.

We fixed this by finding the actual ceiling and building to it on purpose, then owning the moment we broke our own system by pushing changes into a live batch. Both fixes cost us an evening, one of watching zero files complete, one of watching thirty-nine vanish. Neither cost you anything, because by the time you upload your own forty-six PDFs, both evenings had already happened, on our screen, long before yours.