Staff Software Engineer, Tech Foundations
Quick Summary
At least 9 years of software engineering experience, with a proven track record of designing, building, and operating large-scale distributed systems in production environments.
As a Staff Software Engineer, you will shape the technical direction of a large-scale Observability platform supporting critical production systems. You will lead the design, development, and operation of distributed infrastructure that processes millions of telemetry events every second. Working across metrics, logs, and distributed tracing, you will help engineering teams gain deeper insights into system performance, reliability, and operational costs. You will also contribute to building AI-ready observability capabilities, enabling autonomous agents to access operational data safely and efficiently. In this highly collaborative environment, you will influence architectural decisions, mentor engineers, and drive innovation across infrastructure teams. This is an opportunity to solve complex technical challenges while improving developer productivity and system resilience at global scale.
-
Technical Leadership & Architecture: Define the technical vision and architecture for major areas of the Observability platform, leading initiatives from initial design and prototyping through production deployment and ongoing operations.
-
Distributed Systems Development: Design, build, and maintain highly scalable and reliable distributed systems capable of processing tens of millions of events per second.
-
Metrics Infrastructure: Scale and optimize the metrics pipeline using technologies such as OpenTelemetry, vmagent, and Grafana Mimir, while managing infrastructure costs through aggregation, sampling, and unused-metric filtering.
-
Logging & Tracing Modernization: Lead the transition from legacy log aggregation systems to OpenTelemetry-based pipelines and improve log and trace storage capabilities using technologies such as OpenSearch.
-
Query Infrastructure: Develop fast, reliable, and secure query systems for engineers and AI agents, incorporating rate limiting, access controls, isolation, and caching to protect critical incident-response workflows.
-
Telemetry Cost Optimization: Build self-service tools that provide engineering teams with visibility into telemetry costs, including cost attribution by team and service, as well as controls for data granularity, sampling, and retention.
-
Observability & Reliability: Improve incident detection through anomaly detection, automated signal correlation, and enhanced monitoring capabilities.
-
Cross-functional Collaboration: Align technical roadmaps with infrastructure and product teams, influence build-versus-buy decisions, and communicate architectural tradeoffs and investment priorities to engineering leadership.
-
Engineering Excellence: Write high-quality, maintainable, and well-tested code, participate in design reviews, improve operational readiness, and contribute to relevant open-source projects.
-
Mentorship & Knowledge Sharing: Mentor engineers, strengthen technical practices across the team, and help developers instrument services and independently investigate production issues.
Requirements
~2 min read-
Professional Experience: At least 9 years of software engineering experience, with a proven track record of designing, building, and operating large-scale distributed systems in production environments.
-
Technical Leadership: Demonstrated experience leading the technical direction of complex systems or multi-team initiatives, including architecture design, stakeholder alignment, delivery, and production operations.
-
Observability Expertise: Deep experience in at least one major telemetry domain at scale, including metrics platforms such as Prometheus, Grafana Mimir, or VictoriaMetrics; logging technologies such as OpenSearch or Kafka; or distributed tracing solutions such as OpenTelemetry, Jaeger, or ClickHouse.
-
Programming Skills: Strong software development capabilities in Go or Java/Kotlin, with the ability to navigate unfamiliar codebases and contribute to upstream open-source projects.
-
Cloud Infrastructure: Hands-on experience with cloud-native technologies, particularly Kubernetes, AWS, and service mesh solutions such as Istio or Envoy.
-
Communication & Collaboration: Excellent written and verbal communication skills, including the ability to produce clear technical design documents, explain architectural tradeoffs, and present investment recommendations to engineering leadership.
-
Problem-solving: Strong analytical and architectural thinking, with the ability to address complex scalability, reliability, performance, and cost challenges.
-
Open-source Contributions (Preferred): Experience contributing to OpenTelemetry, Prometheus, Grafana Mimir, VictoriaMetrics, OpenSearch, Jaeger, or comparable projects.
-
Advanced Technical Knowledge (Preferred): Familiarity with time-series storage internals, high-cardinality datasets, query optimization, anomaly detection, automated correlation, or SLO-based alerting.
-
Additional Expertise (Preferred): Experience with telemetry cost attribution, large-scale infrastructure cost optimization, AI-agent-facing APIs or tools (including MCP servers), and columnar analytics platforms such as ClickHouse.
What We Offer
~2 min readLocation & Eligibility
About this job
- Pay per year
- $212,000–$265,000 a year
- 26% above the $189,200 median for Software Engineer jobs on Jobera (6,743 salaries).
- Who can apply
- United States
- Posted
- Today
Listing Details
- Posted
- November 3, 2026
- First seen
- November 3, 2026
- Last seen
- November 3, 2026
Posting Health
- Days active
- 0
- Repost count
- 0
- Trust Level
- 80%
- Scored at
- November 3, 2026
Signal breakdown
Similar Software Engineer jobs
View all →Browse Similar Jobs
Stay ahead of the market
Get the latest job openings, salary trends, and hiring insights delivered to your inbox every week.
No spam. Unsubscribe at any time.