Senior Production Engineer

Crusoe is on a mission to accelerate the abundance of energy and intelligence. As the only vertically integrated AI infrastructure company built from the ground up, we own and operate each layer of the stack — from electrons to tokens — to power the world's most ambitious AI workloads. When you join Crusoe, you join a team that is building the future, faster.

We're in the midst of the greatest industrial revolution of our time. The demand for AI compute is boundless, and power is a bottleneck. We're solving that — with an energy-first approach that makes AI infrastructure better for the world and faster for the people innovating with AI.

We're looking for problem-solving, opportunity-finding teammates with a sense of urgency, who believe in the scale of our ambition and thrive on a path not fully paved — people who want to grow their careers alongside a team of experts across energy, manufacturing, data center construction, and cloud services.

If you want to do the most meaningful work of your career, help our customers and partners advance their AI strategies, and be part of a high-performing team that believes in each other, come build with us at Crusoe.

At Crusoe, our Production Engineering team plays a pivotal role in ensuring the reliability and performance of our infrastructure. Production Engineering at Crusoe is dedicated to detecting, analyzing, and preventing issues to maintain high Service Level Agreement through Service Level Indicators (SLIs) and Service Level Objectives (SLOs). Through automation and proactive remediation, our Production Engineers not only resolve common errors automatically but also advise various engineering teams in building resilient code. We prioritize anticipating and resolving issues before they impact our customers, conducting thorough post-mortems, and driving continuous improvement. Our customer-centric approach ensures that clients always have access to the virtual machines they depend on. Join us to help build and maintain the robust systems that power Crusoe's innovative solutions.

As a Production Engineer at Crusoe, your day begins with a review of overnight alerts and system performance metrics to ensure everything is running smoothly. We operate close to the hardware, so many of these will be novel hardware failures on cutting edge equipment that are challenging to diagnose and automate. You will collaborate with your team in a morning stand-up meeting to discuss ongoing projects, recent incidents, and priorities for the day. Your tasks might include automating routine processes, analyzing system logs, and developing tools to enhance our monitoring capabilities. You'll spend part of your day working closely with software engineers, advising on best practices for resilient code and reviewing changes before deployment. Regularly, you will engage in incident response drills, post-mortems, and root cause analysis sessions to learn from past issues and prevent future ones. Throughout the day, you will stay focused on maintaining high SLIs and SLOs, ensuring that our infrastructure remains robust and reliable for our customers. By day's end, you will document your work, share insights with your team, and plan for the next day's challenges, always with a customer-centric mindset.

• 5+ years of professional production engineering experience

• 3+ years of experience contributing to architecture and design (architecture, design patterns, reliability and scaling) of new and current systems

• Bachelor's Degree in Computer Science or related field, or 8+ years relevant work experience

• Solid understanding of infrastructure design, including the operational trade-offs of various designs

• Experience writing high quality code with at least one programming language (Python, Go, or similar)

• Experience with Unix/Linux environments

• Experience with TCP/IP and network programming

• Experience with information security best practices

• Excellent communication skills

• Excited to work with cutting edge AI training & inference hardware and networks

• Experience running large, mission-critical storage systems / NVMe over Fabric

• Experience building with modern infrastructure tools such as Docker, Kubernetes, Ansible, Cloud Formation, Terraform

• Experience building with modern CI/CD practices and build systems, such as GitLab CI/CD, CircleCI, GitHub Actions

• Experience with logging, monitoring and alerting systems and tools