Someone pinged me at 4:40 on a Friday: “need a new Postgres instance, going live Monday, should be quick.” I asked what the read/write ratio looked like and what happens to the checkout flow if the instance falls over for ten minutes. But, they hadn’t thought about either one, because nobody had ever asked them to.
Working with a bad SRE team is frustrating for all the obvious reasons: things break, deploys fail, and nobody answers the page.
It’s 5 PM on a Thursday. Something broke mid-morning and your team has been grinding on it for hours. Good engineers, working hard, coming up empty because the system is poorly documented and the people who built it have either moved on or moved teams. You’ve burned most of the day and you’re no closer to resolution.
So you escalate. You ping the leads. You ask in the channel where someone, surely, knows this service well enough to point you in the right direction.
I was standing outside a conference room watching my team lie to another team about a database outage. It was day four.
Through the glass door, I could see the engineer on the call, explaining with impressive confidence that our cloud provider was having issues. Any minute now, they said, the vendor would resolve it and services would come back up.
I pulled up the provider’s status page. Green across the board. Checked their Twitter. Nothing. I caught the eye of the engineer leading the response and motioned them outside.
Well, maybe not everything.
Look, being a goalie is objectively ridiculous. You strap on forty pounds of equipment designed to protect you from frozen rubber traveling at speeds that would make physicists frown. Then you stand in front of a net and dare people to shoot at you. It’s a strange job.
But here’s the thing: being a goalie is basically the same job as running ops, or security, or honestly, any part of software development where you’re the one who has to keep the thing from breaking.
Photo by Patrick Konior on Unsplash
We need to talk about how we ship software. Not because we’re doing it wrong, but because we can do it so much better, for our customers, for our colleagues, and for ourselves.
Here’s the shift we need to make: deploying code and releasing features are not the same thing, and they shouldn’t happen at the same moment.