7 things that stuck after rebuilding my always-on agent box

i'm going to show you the seven things that survived rebuilding a linux box that runs coding agents around the clock, so you can skip the versions of it that didn't work

most people run their agents on the laptop they're sitting at. that's fine until the first long run, and then you start building a box, and then you rebuild it. i've done that more times than i'd like to admit

specifically you're going to walk away with all of this:

why your image probably isn't pinned even though you pinned it

the deploy that reports success & silently rolls back 95 seconds later

where the weight in your image actually is, which is never where you think

the update that only ever reaches new users & never your oldest ones

let's get into it

1. pin by digest, not by tag

i thought my image was pinned for a month. it was pinned to a tag, which anyone with push access can move underneath you. that's a bookmark, not a pin

the fix is boring: resolve the tag once, take the sha, and reference that. every box built after it is built from an exact thing rather than whatever was there this morning

2. exit code 0 is not success

i shipped a 13.6gb image. the deploy reported success. flyctl exited 0. everything looked completely fine

95 seconds later the machine had quietly reverted to the previous image, and the only trace anywhere was one line in the machine's own event list. nothing in the output, nothing in the exit code

exit 0 means the api accepted your request. it does not mean your machine is running your code. read the events after a deploy

3. measure per layer, never in total

when i went looking for the weight, it was nowhere near where i'd have guessed

one npm package: 1.4gb on its own. pytorch: 754mb, on a machine with no gpu, already the cpu-only build. npm globals across everything: 3.2gb

you can stare at a total for an hour & learn nothing. one du per layer & it's obvious in a minute

4. move things, don't copy them

my entrypoint copied a home skeleton onto the volume instead of moving it. 867mb of pure duplication, on every single box, for months

that one was free to fix & i'd shipped it dozens of times without noticing

5. an update has to reach disks that already exist

this is the one i'd most want you to take away. my image updates only ever helped NEW boxes. anyone who'd been with me longest was running the oldest tools, which is exactly backwards

if your users have persistent volumes, an image update is not an update until something additively syncs the new content onto the disks already out there

6. one flag should never arm two decisions

for a while i had auto-provisioning & disk deletion behind the same switch. so "a payment should build a machine without me watching" also silently meant "a cron may erase someone's only copy of their work without me watching"

those are not the same risk appetite & a single flag can't express disagreeing with one of them

7. you pay for uptime, not for work

an agent blocked on an api call bills exactly the same as an agent compiling. so an overnight run is billed for the whole night, mostly for waiting

suspend when idle helps a lot & saves you nothing during the part you actually care about, because that's the part where it's running

that's the list

none of it is clever. all of it cost me a rebuild, & most of it cost me one twice

i sell the version where somebody else does this, & i'm obviously biased. if you'd rather do it yourself, the seven above are the whole map i wish i'd had