Incident Management & Resilience Lead

Incidents get resolved, but the fixes that would stop them recurring slip down the backlog — and a few weeks later, a version of the same failure comes back. Most teams live with that. On a platform as critical as Buildkite's, we've decided not to, which is why we're creating a role whose whole remit is to own how we respond to incidents and design them out for good.

Buildkite runs production builds for teams like OpenAI, Anthropic, Uber, Shopify, Airbnb, Canva and Pinterest — software delivery in the critical path for over a billion daily users. When something breaks here, it breaks loudly, for people who can't afford it to.

Until now, Incident response has been carried by whoever was closest, in the gaps between everything else they own. It's held up. But "held up" is no longer the bar for a platform this critical. We'll be asking you to set the standard for how the whole engineering org detects, responds to, learns from, and designs out the incidents that matter — end to end.

You'd own incident management across engineering — the definition, process, tooling, and adherence — and make it stick.

In the moment, you're in the room on the high-impact incidents, driving clarity, coordination and a clean resolution. Between incidents — where the real work lives — you're running postmortems that go somewhere, making sure the followups get done every time, and turning what each one taught us into the changes that make the next incident smaller.

You'd work across every product team plus Security and Support, holding one consistent resilience bar and bringing engineering managers with you. And you'd coach engineers on how to run an incident, communicate through it, and own the outcome.

You've built or materially lifted incident management capability somewhere before, and you've got a clear, earned point of view on what good looks like.

You can read the code, dig into the telemetry, and lead a real investigation — not just chair the call. You're calm when it's sustained and messy: you make incidents smaller, not louder. And you can hold a firm bar across teams that don't report to you without softening the standard or bruising the relationship — the diplomacy to find common ground and the spine to keep the line.

You'll be at home with observability tooling (Datadog, Honeycomb), AWS, Terraform, and the failure modes distributed systems throw up at scale. Our stack is Ruby on Rails, PostgreSQL, Kafka and Redis on AWS — you won't need all of it on day one.

The one thing we won't move on: you've owned incident response for production systems at real scale for a number of years, meaning you've experienced a breadth of different challenges and scenarios. It's this knowledge and real world experience we're hoping to benefit from. 

• Would rather draw the map on a blank page than inherit someone else's
• Get more out of the incident that never happened than the heroic save
• Can hold a standard across teams you don't manage, and keep the relationship
• Read "flat, high-autonomy, little scaffolding" as room to move, not risk

• Want a mature incident function to run, not one to build from the ground up
• Would rather chair the call than open the code and dashboards yourself
• Need top-down authority before people will move for you
• Find cross-team persuasion draining rather than part of the fun

• First of its kind here. No one has owned this before you. You define what incident management at Buildkite means — the mandate is real and yours to shape.
• Scale that's genuinely rare. Billions of builds, in the critical path for some of the strongest engineering teams on the planet — systems that run continuously and can't be casually restarted.
• Ownership, not tickets. Flat structure, high autonomy. You're trusted to make the calls that matter and own the result.
• Remote, properly. We've worked this way since 2013 — async and built for deep focus, with genuine overlap across our ANZ and US-Pacific teams.
• Small enough that it counts. ~150 people. What you change is visible across the org.

Our engineering teams are based across ANZ and US-Pacific time zones — a conscious decision that lets us move quickly with real overlap and minimise fully async work. So while Buildkite is fully remote, we don't hire everywhere: if you're applying from outside these regions, we're not currently in a position to hire you, and can't offer sponsorship.

Every application gets a response. We're wired for velocity, so you won't be left waiting on us. If this is the problem you've been wanting to get your hands on, apply — or reach out with questions first.

At Buildkite, we value diversity and celebrate all types of skills, backgrounds, and experiences. We’re dedicated to fostering an inclusive environment and providing reasonable accommodations throughout our recruitment process.

If you need any accommodations or support during the application or interview process, please reach out to us at accommodations@buildkite.com.