steinborn.dev
← all notes
ai-engineering

The deploy that only half happened

· 2 min read · Benjamin Steinborn

Recently, I rebuilt LibrisVault’s frontend while the service was running. The build succeeded, the files landed in dist/, but the dashboard promptly broke: blank page, console full of 404s for asset files that were sitting right there on disk. The server was serving from the directory the files were in and could not find them.

The server remembers boot, not disk

The cause: the static file plugin was configured with wildcard: false. In that mode fastify-static walks the directory at startup and registers one route per file that exists at that moment. It is a boot-time snapshot of the filesystem, not a live view.

The process behaved like a waiter working from the menu printed at the start of the shift: dishes the kitchen decided to add since are real and ready but still impossible to order.

The production frontend build writes content-hashed filenames - index-a1b2c3.js becomes index-d4e5f6.js. The rebuild swapped the hashes on disk, but the running process still owned the route table from its own boot: old hashes routed, new hashes did not exist as routes, so real files resulted in 404 errors. Nothing was corrupted. The process and the disk simply described two different points in time.

One restart later, everything worked, the fix was never the mystery. The lesson was.

Deploys end in process state, not on disk

“The files are there” is not the same claim as “the service serves them”. Servers cache the world at boot in more places than route tables: parsed configs, schema snapshots, plugin registrations, connection settings. Any of them can turn a file swap into a half-deploy where disk says version two and the process still lives in version one.

A deploy is finished when the process state matches the disk, which puts restarts inside the deploy rather than in the troubleshooting that follows it. Swap artifacts atomically, restart deliberately and you will know which of your server’s views of the world were taken at boot. In LibrisVault the lesson is written down where it will be found: the installer ends on a restart, and the troubleshooting guide names this failure by its symptom.