ML Platform Engineer, Distributed Training

Join a high-impact AI engineering team supporting healthcare and life sciences applications through large-scale foundation model training. As an ML Platform Engineer specializing in distributed training, you will work across GPU infrastructure and model training workloads on a Google Kubernetes Engine (GKE) platform powered by H200 and GB200 GPUs. You’ll collaborate with infrastructure engineers and research scientists to optimize training performance, diagnose complex failures, and improve multi-node execution at scale. Rather than focusing solely on cluster operations, you’ll take end-to-end ownership of the training stack, from workload scheduling and GPU networking to benchmarking and model evaluations. Your work will help scientists train models more efficiently while improving system reliability, scalability, and cost effectiveness. This role offers a hands-on engineering environment where technical depth, proactive problem-solving, and clear communication are essential.

• Professional experience: At least 5 years of engineering experience, including recent hands-on work with GPU training infrastructure and large-scale model training.

• Python engineering: Strong production-level Python skills, including writing clean, maintainable code, automated tests, participating in code reviews, and following sound software development practices.

• Kubernetes and GKE: Practical experience with Kubernetes on Google Cloud, including GPU node pools, job scheduling and queueing systems such as Kueue, and resource quota and capacity management.

• Distributed training expertise: Direct experience running and debugging multi-node PyTorch training jobs and working with Ray Train.

• GPU networking and interconnects: Strong understanding of RDMA, GPUDirect, NCCL tuning, and multi-node GPU architectures involving H200 or GB200 hardware.

• Performance analysis: Demonstrated ability to design and execute training benchmarks, evaluate model performance, identify bottlenecks, and use empirical evidence to guide optimization.

• Technical depth: Ability to explain system architecture, implementation choices, and performance behavior in detail, demonstrating independent engineering judgment rather than relying on AI coding tools to supply the underlying reasoning.

• Ownership and communication: A track record of accountability, proactive problem-solving, transparent communication, and effective collaboration in demanding engineering environments.

• Preferred domain experience: Familiarity with healthcare or life sciences data, including genomics, clinical, or biomedical information, and experience working under HIPAA requirements.

• Additional preferred skills: Experience building observability and telemetry for training platforms, including metrics, tracing, alerting, and end-to-end pipeline testing.