Staff Site Reliability Engineer - Dev Platform

BrazilRemoteFull-timelead

EngineeringDevops Engineer

1 views0 saves0 applied

Apply Now

Quick Summary

Overview

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Staff Site Reliability Engineer – Dev Platform based in Brazil.

Technical Tools

EngineeringDevops Engineer

This is an opportunity to shape the reliability and scalability of a critical internal developer platform within a global financial technology environment. You will take ownership of an AI Gateway that provides engineering teams with governed, self-service access to leading large language models (LLMs), helping optimize performance, resilience, security, and cost. As a senior technical leader, you will define reliability standards, guide architectural decisions, and ensure the platform can support growing AI workloads across the organization. You will work across engineering, security, finance, and AI governance to build robust infrastructure and effective operational practices. The role combines hands-on site reliability engineering with strategic technical leadership, incident management, and continuous improvement. It is ideal for an experienced infrastructure professional passionate about cloud-native systems, AI platform engineering, and enabling developers to build and deliver software more efficiently.

Own the reliability, availability, and performance of the production AI Gateway, defining and maintaining service-level objectives (SLOs), error budgets, and on-call practices.
Design and implement resilience mechanisms, including multi-provider model routing, failover, retries, rate limiting, and graceful degradation when external model providers experience disruptions.
Set technical direction for gateway architecture, capacity planning, and scaling to accommodate increasing LLM traffic and expanding workloads.
Develop comprehensive observability for AI traffic, including latency, errors, token consumption, and cost attribution by team and use case.
Collaborate with Security, Finance, and AI governance stakeholders to implement access controls, usage budgets, audit logging, and appropriate data-handling policies.
Lead incident response and conduct blameless post-incident reviews, translating findings into long-term reliability improvements.
Automate infrastructure and operational processes through infrastructure as code, safe deployment strategies, and progressive rollouts.
Support secure and reliable integrations with LLM providers and internal AI services.
Mentor engineers, establish engineering best practices, and raise operational standards across the developer platform organization.
Influence technical decisions across teams through clear design documentation, architectural guidance, and collaborative leadership.

Requirements

~2 min read

Extensive experience in site reliability engineering, infrastructure engineering, or platform engineering, with a proven track record of operating high-traffic, business-critical production services.
Deep hands-on experience with Kubernetes, a major cloud provider, and infrastructure as code tools such as Terraform.
Strong understanding of SLOs, observability, incident management, and reliability engineering principles, with the ability to prioritize improvements based on business impact.
Proficiency in a programming language such as Rust or Python, along with the ability to read, troubleshoot, and contribute to production codebases.
Experience with API gateways, reverse proxies, or traffic management systems, including load balancing, rate limiting, failover, and caching.
Practical experience with Consul and Vault for service discovery, configuration management, and secrets management.
Knowledge of authentication and authorization mechanisms for internal platforms and APIs, including OAuth 2.0/OIDC, JWT, mutual TLS (mTLS), and API keys, particularly in multi-tenant environments.
Experience with model evaluation systems, including building or operating evaluation pipelines to compare models, prompts, and agents and identify quality regressions before and after deployment.
Familiarity with coding agents and agent harnesses, such as OpenHands, Claude Code, or comparable tools, including safe execution in sandboxed environments.
Experience with agent memory systems and supporting data stores, such as vector databases, search systems, relational databases, and key-value stores, with an emphasis on reliable operations.
Staff-level technical leadership, including the ability to establish direction across teams, write effective design documents, and build alignment around technical decisions.
Strong judgment when balancing reliability, infrastructure costs, and developer productivity.

Nice to have:

Experience operating LLM gateways or integrating LLM provider APIs at scale, including streaming, token-based rate limiting, model routing, and caching.
Experience hosting and serving models using technologies such as vLLM, or operating distributed inference and training workloads with Ray, including GPU scheduling, batching, autoscaling, and cost-per-token optimization.
Experience designing, developing, and maintaining Kubernetes controllers and operators using tools such as controller-runtime or Kubebuilder, including custom resource definitions, reconciliation loops, and safe upgrades.
Familiarity with cost attribution, budgeting, or FinOps practices for usage-based services.
Experience working in security-sensitive or regulated environments, such as financial services or cryptocurrency.
Contributions to open-source projects involving gateways, proxies, or observability.