Posts for: #Sre

Brains in Jars Don’t Have a Failover Plan

Brains in Jars Don't Have a Failover Plan

It was nearly midnight on a Friday, and I was the one person finishing the work behind a sales deal our rep had called huge for the company. My manager was a few feet away, scrolling something on his phone, a kind of show of solidarity, when it rang. He put it on speaker and set it down on the desk between us. The sales rep wanted to know if the work would be done by Monday morning, like he’d promised the client it would be. My manager didn’t check with me. He didn’t ask what was left, or what the weekend would cost me to get there. He just said it, loud enough that I heard every word from across the room: “I don’t care how long it takes, you will get it done for Monday morning.” I think he hung up proud of himself.

Read more →

Why Good SREs Feel Like Wall Builders

Why Good SREs Feel Like Wall Builders

Someone pinged me at 4:40 on a Friday: “need a new Postgres instance, going live Monday, should be quick.” I asked what the read/write ratio looked like and what happens to the checkout flow if the instance falls over for ten minutes. But, they hadn’t thought about either one, because nobody had ever asked them to.

Working with a bad SRE team is frustrating for all the obvious reasons: things break, deploys fail, and nobody answers the page.

Read more →

Why Doesn’t Anyone Own That?

It’s 5 PM on a Thursday. Something broke mid-morning and your team has been grinding on it for hours. Good engineers, working hard, coming up empty because the system is poorly documented and the people who built it have either moved on or moved teams. You’ve burned most of the day and you’re no closer to resolution.

So you escalate. You ping the leads. You ask in the channel where someone, surely, knows this service well enough to point you in the right direction.

Read more →

The Lie That Taught Me About Blameless Culture

I was standing outside a conference room watching my team lie to another team about a database outage. It was day four.

Through the glass door, I could see the engineer on the call, explaining with impressive confidence that our cloud provider was having issues. Any minute now, they said, the vendor would resolve it and services would come back up.

I pulled up the provider’s status page. Green across the board. Checked their Twitter. Nothing. I caught the eye of the engineer leading the response and motioned them outside.

Read more →

Everything I learned about Ops, I learned playing goal

Well, maybe not everything.


Look, being a goalie is objectively ridiculous. You strap on forty pounds of equipment designed to protect you from frozen rubber traveling at speeds that would make physicists frown. Then you stand in front of a net and dare people to shoot at you. It’s a strange job.

But here’s the thing: being a goalie is basically the same job as running ops, or security, or honestly, any part of software development where you’re the one who has to keep the thing from breaking.

Read more →