Staff Software Engineer, Backend (Observability)

CanadaRemoteFull-timelead

EngineeringBackend Developer

0 views0 saves0 applied

Apply Now

Quick Summary

Key Responsibilities

Own the architecture of the telemetry pipeline, balancing reliability, query performance, and infrastructure costs at a scale exceeding 200 TB of daily data.

Requirements Summary

At least 8 years of experience designing, developing, and launching large-scale backend systems using languages such as Python or Kotlin. Extensive exper

Technical Tools

EngineeringBackend Developer

The Staff Software Engineer will serve as the technical lead for a large-scale observability platform supporting hundreds of services and engineering teams. You will shape the architecture and long-term strategy for telemetry infrastructure processing more than 200 TB of metrics, logs, and traces every day. The role focuses on improving system reliability, query performance, cost efficiency, and the quality of engineering insights. You will lead initiatives involving structured events, intelligent log sampling, and AI-assisted development practices. Combining hands-on backend engineering with technical leadership, you will mentor a team of six engineers and drive cross-functional projects. This is an opportunity to influence engineering standards and build foundational infrastructure within a remote-first environment.

You will own the technical direction and evolution of the observability platform, ensuring engineers have reliable, efficient tools to monitor systems and investigate incidents. Your responsibilities will include:

Own the architecture of the telemetry pipeline, balancing reliability, query performance, and infrastructure costs at a scale exceeding 200 TB of daily data.
Design and implement log sampling strategies that reduce data volume and operational expenses while preserving the information needed for effective incident investigation.
Lead the transition toward wide, context-rich events that improve observability and reduce noise during debugging.
Shape the integration of observability practices into AI-assisted and agentic software development, ensuring coding agents and code review processes identify instrumentation gaps before deployment.
Define the team's technical strategy and roadmap over a long-term planning horizon, connecting engineering priorities to business-critical initiatives.
Lead major technical projects and collaborate with Product Management, Design, Analytics, Infrastructure, and other engineering teams throughout the product development lifecycle.
Establish technical solutions, engineering standards, and operational processes that improve team effectiveness and system sustainability.
Take ownership of service availability and operational readiness by implementing appropriate monitoring, alerting, testing, incident triage rotations, and playbooks.
Define and promote code review and system design standards, sharing best practices through technical documentation, written guidance, and engineering presentations.
Mentor and develop a team of six engineers through technical guidance, constructive feedback, and hands-on leadership.
Advocate for observability standards and drive their adoption across multiple engineering teams.

Requirements

~2 min read

The ideal candidate is an experienced backend engineer with a strong background in distributed systems and large-scale telemetry infrastructure. They should combine deep technical expertise with the ability to establish engineering standards, influence cross-functional teams, and develop other engineers. Key qualifications include:

At least 8 years of experience designing, developing, and launching large-scale backend systems using languages such as Python or Kotlin.
Extensive experience building and operating highly available distributed systems.
Strong knowledge of technologies such as AWS, MySQL, Flink, and Kubernetes.
Hands-on experience building or operating high-volume telemetry systems involving metrics, logs, and/or traces.
Familiarity with observability technologies and infrastructure tools such as OpenTelemetry, Kafka, and Terraform.
Strong understanding of modern observability practices, including structured and wide events, log sampling, cardinality management, and Service Level Objectives (SLOs).
Proven ability to drive the adoption of platform standards across multiple engineering teams through effective tooling, communication, and collaboration.
Experience leading major technical initiatives, delivering significant system components, and planning the deprecation of existing functionality.
Ability to develop high-quality, maintainable code and establish standards that make systems easier for other engineers to understand and use.
Comfort navigating ambiguity and moving between low-level programming details and large-scale system architecture.
Demonstrated ability to gather, incorporate, and act on feedback from engineering and cross-functional peers.
Strong written and verbal communication skills, with the ability to collaborate effectively across a global engineering organization.
Ability to provide technical leadership, mentor engineers, and influence engineering practices beyond your immediate team.