Senior Software Engineer, SRE and Production Engineering - DGX Cloud

CanadaRemoteFull-timesenior

Software EngineerSoftware Engineering

0 views0 saves0 applied

Apply Now

Quick Summary

Key Responsibilities

Develop automation for bare-metal server provisioning, hardware validation, firmware and software upgrades, repairs, and cluster lifecycle management.

Requirements Summary

At least 12 years of experience building software for or operating production infrastructure, including substantial hands-on bare-metal experience. Strong programming skills in Go or Python,

Technical Tools

Software EngineerSoftware Engineering

Join a team building and operating large-scale GPU infrastructure that powers demanding artificial intelligence workloads. As a Senior Software Engineer specializing in Site Reliability Engineering (SRE) and Production Engineering, you will develop the automation and operational systems that transform bare-metal hardware into reliable, production-ready cloud services. You will work hands-on with advanced GPU servers, data processing units, firmware, Linux, and Kubernetes across cloud partner and on-premises environments. Your work will improve infrastructure provisioning, hardware validation, system recovery, and cluster lifecycle management. You will also help strengthen production reliability by investigating complex failures and turning recurring incidents into lasting automated solutions. This remote opportunity offers the chance to contribute to next-generation AI infrastructure at significant scale.

You will design, build, and maintain the software and operational tooling required to deploy, manage, and support large-scale GPU infrastructure. Your main responsibilities will include:

Develop automation for bare-metal server provisioning, hardware validation, firmware and software upgrades, repairs, and cluster lifecycle management.
Build tools that interact with Baseboard Management Controller (BMC) and Redfish interfaces to monitor hardware health, manage server states, and support recovery workflows.
Operate and advance NVIDIA NVL72 systems and BlueField-3 or newer Data Processing Units (DPUs) across cloud partner and on-premises environments.
Diagnose and resolve complex failures involving servers, DPUs, GPU and CPU systems, networking, Linux, and Kubernetes.
Convert recurring infrastructure issues into automated detection, troubleshooting, and repair processes.
Define validation procedures and production handoff criteria to ensure new infrastructure capacity is introduced safely and consistently.
Participate in on-call rotations, incident response, root-cause investigations, and follow-up activities to implement permanent corrective solutions.
Collaborate with hardware, networking, platform engineering, data center operations, and partner teams to resolve issues across organizational boundaries.
Improve infrastructure reliability, operational consistency, and automation across the production fleet.

Requirements

~1 min read

The ideal candidate brings extensive software engineering and production infrastructure experience, with strong hands-on expertise in bare-metal systems and large-scale hardware operations. The following qualifications and capabilities are expected:

At least 12 years of experience building software for or operating production infrastructure, including substantial hands-on bare-metal experience.
Strong programming skills in Go or Python, with a proven track record of delivering production automation and services.
Direct experience using BMC and Redfish interfaces for server provisioning, hardware health inspection, power management, or fault diagnosis.
Practical experience working directly with NVIDIA GPU hardware, particularly NVL72 systems, and BlueField-3 or newer DPUs.
Strong knowledge of Linux, firmware and driver management, network boot processes, and server lifecycle operations from initial provisioning through repair.
Experience maintaining production reliability through on-call responsibilities, incident response, observability, and implementation of durable solutions.
Ability to troubleshoot complex issues across hardware, host operating systems, networking, and distributed services.
Strong communication skills and demonstrated ownership of technical problems involving multiple teams.
Bachelor's or Master's degree in Computer Science, or equivalent practical experience.

Experience operating BlueField DPUs in DPU mode, including host-to-DPU connectivity and lifecycle troubleshooting.
Familiarity with NVLink, InfiniBand, Spectrum-X, or GPU cluster performance validation.
Experience designing safe and repeatable workflows for rack-scale deployment, firmware upgrades, hardware replacement, and customer handoffs.
Knowledge of Kubernetes, GitOps, Argo CD, Service Level Objectives (SLOs), and fleet-wide automation.