Principal Model Optimization Engineer

ML Platform at Roblox supports hundreds of ML use cases and billions of inferences per day across Discovery, Safety, Engine and much more. We build what ML engineers use to go from idea to production: data and training pipelines, the SDKs and tools they build with, the GPU fleet and the scheduling that runs their jobs.

As a Principal Engineer on ML Platform, you will set the technical direction for these systems. You will find what slows ML engineers down or wastes compute and design durable fixes. You will measure the results and keep improving them until they show up as faster iteration, more reliable jobs and better use of every GPU.

• Design platform capabilities that many teams adopt, so that one investment improves hundreds of use cases instead of one.
• Build GPU and compute management that keeps a scarce, mixed fleet busy. This covers quotas and priority tiers, preemption, scheduling across clusters, and capacity planning with Infrastructure.
• Define the platform’s success metrics, instrument them, and improve them quarter over quarter. Examples include time from code to first training run, job success rate, GPU utilization, and cost per training run or per inference.
• Lead deep investigations when jobs are slow or failing, from pipeline and scheduler behavior down to GPU profiles and hardware errors. Turn the findings into fixes that stay fixed.
• Work closely with ML engineers and data scientists to learn how they work, and make the paved path the easiest one to take.
• Partner with Infrastructure, Data Platform and Security on shared foundations such as storage, networking, data access and compliance.
• Raise the technical bar through design reviews, written proposals and code, and mentor senior engineers.

• 6+ years of professional experience and a tool chest of system design experience upon which to draw to build performant systems for all of Roblox.
• A track record of building ML infrastructure end to end. Examples include training pipelines, workflow orchestration (e.g. Kubeflow, Airflow, Ray), data or feature pipelines, experiment tracking and model registries, or model serving.
• Experience designing SDKs, APIs or developer tools that were widely adopted, and a strong sense of what makes a platform easy to use.
• Hands-on experience running GPU workloads at scale on Kubernetes. That includes scheduling, quotas, multi-cluster dispatch, autoscaling, and debugging GPU and node failures (e.g. Xid errors, NCCL hangs).
• Enough depth in training and inference performance (distributed training, mixed precision, batching, quantization) to know where the platform should do the work for its users.
• A habit of measuring: you set clear success metrics, ship in steps, and keep iterating until the numbers move.
• Experience leading cross-team technical work without direct authority, and writing clearly for both engineers and leaders.
• A Bachelor’s degree in Computer Science, Computer Engineering or a related field, or equivalent experience.