Staff Machine Learning Engineer

Toast creates technology to help restaurants and local businesses succeed in a digital world, helping business owners operate, increase sales, engage customers, and keep employees happy.

The Machine Learning Platform team builds and operates the core infrastructure that powers ML across Toast — the feature store, model hosting and serving, the experimentation platform, training pipelines, and the tooling ML engineers and data scientists rely on every day. Our work directly enables the models that drive personalization, forecasting, fraud detection, search, and the growing set of AI-powered experiences shipping to restaurants.

Toast is seeking a Staff Software Engineer to act as a technical leader on the ML Platform team, shaping the systems that will carry Toast's ML capabilities into the next decade. The role involves driving architectural direction across the platform, delivering foundational infrastructure that other teams build on, and elevating fellow engineers. The ideal candidate is a domain expert who partners with ML engineers, data scientists, product, and infrastructure leadership on high-leverage opportunities.

This position suits an engineer comfortable writing production code, leading technical design for distributed systems, and influencing organizational decisions about how Toast builds and deploys ML.

• →Own technical direction of the ML Platform — feature store, model hosting and serving, experimentation, training infrastructure — driving architectural decisions around scalability, reliability, latency, and cost
• →Lead design and delivery of large-scope platform initiatives from conception through production, coordinating across ML, data, and infrastructure teams
• →Identify and resolve systemic technical challenges: online/offline feature parity, model deployment friction, experimentation velocity, GPU utilization, cross-team dependencies
• →Set and maintain a high engineering quality bar through hands-on code contributions, design reviews, and mentorship of platform and ML-adjacent engineers
• →Partner with ML engineering, data science, product, and platform leadership to translate ML strategy into technical roadmaps
• →Define the paved paths ML teams use to ship models safely — from feature registration through canary rollout, monitoring, and rollback
• →Leverage AI-augmented development tools to increase development velocity and code quality

• 8+ years delivering complex backend or infrastructure systems at scale
• Direct experience building or operating core ML infrastructure — feature stores, model serving, experimentation platforms, training orchestration, or equivalent
• Mastery of a modern backend language such as Python, Java, Kotlin, Go, or Scala
• Deep proficiency with distributed systems concepts: consistency, latency, throughput, fault tolerance, and observability
• Strong understanding of data modeling, query languages, and the online/offline data patterns that underpin ML systems
• Demonstrated technical leadership, with ability to drive cross-team alignment and influence engineering, product, and business stakeholders
• Bachelor's degree in Computer Science or a related field, or equivalent practical experience

• Hands-on experience with open-source or commercial ML platform components (e.g. Tecton, MLflow, SageMaker, Databricks)
• Experience building or operating experimentation / A-B testing platforms at scale
• Familiarity with real-time streaming systems (Kafka, Flink, Spark Streaming) and their use in feature computation
• Experience serving LLMs or large deep-learning models in production, including GPU capacity planning and inference optimization
• Comfort with Kubernetes and modern cloud-native infrastructure
• Prior work supporting internal-developer-facing platforms with a product mindset

At Toast, one of our company values is that we're hungry to build and learn. We believe learning new AI tools empowers us to build for our customers faster, more independently, and with higher quality. We provide these tools across all disciplines, from Engineering and Product to Sales and Support, and are inspired by how our Toasters are already driving real value with them. The people who thrive here are those who embrace changes that let us build more for our customers; it’s a core part of our culture.

At Toast, our employees are our secret ingredient—when they thrive, we thrive. The restaurant industry is one of the most diverse, and we embrace that diversity with authenticity, inclusivity, respect, and humility. By embedding these principles into our culture and design, we create equitable opportunities for all and raise the bar in delivering exceptional experiences.

We Thrive Together

We embrace a hybrid work model that fosters in-person collaboration while valuing individual needs. Our goal is to build a strong culture of connection as we work together to empower the restaurant community. To learn more about how we work globally and regionally, check out: https://careers.toasttab.com/locations-toast.

Toast is committed to creating an accessible and inclusive hiring process. As part of this commitment, we strive to provide reasonable accommodations for persons with disabilities to enable them to access the hiring process. If you need an accommodation to access the job application or interview process, please contact candidateaccommodations@toasttab.com.

------

For roles in the United States, it is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.