Posts for: #Reliability

Brains in Jars Don’t Have a Failover Plan

Brains in Jars Don't Have a Failover Plan

It was nearly midnight on a Friday, and I was the one person finishing the work behind a sales deal our rep had called huge for the company. My manager was a few feet away, scrolling something on his phone, a kind of show of solidarity, when it rang. He put it on speaker and set it down on the desk between us. The sales rep wanted to know if the work would be done by Monday morning, like he’d promised the client it would be. My manager didn’t check with me. He didn’t ask what was left, or what the weekend would cost me to get there. He just said it, loud enough that I heard every word from across the room: “I don’t care how long it takes, you will get it done for Monday morning.” I think he hung up proud of himself.

Read more →

Why Good SREs Feel Like Wall Builders

Why Good SREs Feel Like Wall Builders

Someone pinged me at 4:40 on a Friday: “need a new Postgres instance, going live Monday, should be quick.” I asked what the read/write ratio looked like and what happens to the checkout flow if the instance falls over for ten minutes. But, they hadn’t thought about either one, because nobody had ever asked them to.

Working with a bad SRE team is frustrating for all the obvious reasons: things break, deploys fail, and nobody answers the page.

Read more →