Senior Software Engineer - Orchestration

As a Senior Software Engineer on the Orchestration pod within Engine Productivity, you'll design and run the platform that executes large-scale end-to-end and integration tests, running the real, shipping client on real hardware, across Roblox's data centers, cloud, and our own device labs, so our engineering teams can ship the engine, clients, Studio, and more with speed and confidence.

Every Roblox engine, client, and Studio change, along with the experiences built on top of them, should ship with confidence, and the Orchestration team is the layer that makes that possible. We build large-scale distributed services that turn thousands of test suites into a reliable, push-button pipeline: fanning work out across fleets of machines and real devices, moving artifacts to where they're needed, managing single- and multi-client test state, and giving test owners and maintainers a system to validate their own runs. It looks a lot like building a specialized cloud platform, with capacity-aware scheduling, isolation and sandboxing, and smart retry and backoff, plus the classic distributed systems problems (fairness, efficiency, failure handling, and reliability) at Roblox scale.

• Design and build the distributed services at the heart of our test platform: schedulers, control-plane components, and APIs that fan end-to-end and integration test work across fleets of machines and real devices.
• Run the real, retail-configuration client on physical hardware at scale, coordinating single- and multi-client test state, join flows, and gameplay scenarios.
• Build capacity-aware scheduling, isolation/sandboxing, and smart retry and backoff to run tests reliably and efficiently.
• Build systems that track the health of our hardware and cloud fleet and automatically detect and recover from failures with minimal human intervention.
• Develop the pieces that move artifacts, manage client state, and let test owners validate their own runs, including the launcher that drives the game client.
• Help evolve our legacy automation harnesses and point solutions into a cohesive Universal Test Platform, defining the core APIs other Client teams build on.
• Take projects from design through production, and support the many engineering teams (engine, clients, Studio, and beyond) that grow to depend on the platform.

• Strong experience building distributed systems or backend services such as schedulers, control planes, orchestration, or similar large-scale infrastructure.
• Proficiency in Go and/or Python (we work mostly in Go and Python, with some Luau for test and integration tooling).
• A solid grasp of distributed systems fundamentals: fairness, efficiency, failure handling, and reliability.
• Experience building systems that stay healthy on their own: health monitoring, fault detection, and automated recovery.
• Familiarity with running workloads on physical test hardware (phones, computers, consoles) is a strong plus; comfort getting hands-on with real devices.
• Experience with end-to-end or integration testing of real applications, not just unit tests or standard CI pipelines, is a plus.
• A passion for correctness and doing things "the right way."