2h ago

↻ Repost

$50,000 to $80,000 USD/yr

Senior Data Engineer

BrazilRemoteFull-timesenior

Data EngineerData

1 views0 saves0 applied

Apply Now

Quick Summary

Requirements Summary

At least 7 years of professional experience in data engineering, including a minimum of 3 years operating production-grade streaming data pipelines.

Technical Tools

Data EngineerData

We are looking for an experienced Senior Data Engineer to take ownership of a next-generation data platform built around real-time streaming and modern lakehouse technologies. In this role, you will design, develop, and operate scalable data pipelines that transform operational data into reliable, business-critical data products. You will work extensively with Apache Spark Structured Streaming, Change Data Capture (CDC), Apache Iceberg, and AWS infrastructure to deliver data transformations in near real time. Beyond pipeline development, you will oversee compute infrastructure, optimize performance and costs, and ensure the reliability of production systems. You will also build observability tools and improve developer workflows, enabling engineering teams to onboard and manage their own data models. This is an opportunity to influence the evolution of a large-scale data ecosystem while solving complex technical challenges in a distributed, international environment.

Streaming Data Pipeline Development: Design, build, and maintain robust Apache Spark Structured Streaming applications that process Change Data Capture (CDC) events into reliable, near-real-time data models.

Stream Processing Reliability: Implement and optimize stateful processing, slowly changing dimensions (SCD), checkpointing, state stores, late-arriving and partial records, and safe recovery mechanisms to ensure accurate and consistent data processing.

AWS Infrastructure Management: Own and scale AWS EMR clusters, managing instance sizing, resource allocation, managed scaling, multi-cluster deployments, driver and executor memory optimization, and workload consolidation to improve efficiency and reduce infrastructure costs.

Lakehouse Management: Maintain and optimize Apache Iceberg tables at scale by designing partitioning strategies and implementing compaction, snapshot expiration, and delete-file cleanup, including specialized maintenance for high-churn tables.

Data Source Integration: Onboard new business domains onto the streaming platform by configuring CDC sources and destinations through Estuary Flow, setting up Kafka-compatible topics, validating backfills in Snowflake, and coordinating with source database owners.

Production Troubleshooting: Investigate and resolve complex production incidents across the data platform, addressing issues such as connection failures, offset regressions, token expiration, schema inconsistencies, and write conflicts while focusing on root-cause resolution.

Monitoring and Observability: Develop monitoring capabilities and diagnostic tools to improve incident detection and resolution, including integrating Spark driver and executor logs with Grafana Loki, implementing watchdog mechanisms, and creating AI-assisted operational runbooks.

CI/CD and Developer Experience: Improve deployment workflows, configuration management, automated pipelines, and operational safeguards to enable engineering teams to independently and safely deploy new data models.

Cross-Team Collaboration: Support partner engineering teams with ad hoc data ingestion, backfill requests, technical escalations, and platform-related guidance.

Requirements

~2 min read

At least 7 years of professional experience in data engineering, including a minimum of 3 years operating production-grade streaming data pipelines.

Apache Spark Expertise: Strong hands-on experience with Apache Spark, particularly Structured Streaming, state stores such as RocksDB, checkpointing, stateful processing, and performance optimization.

Lakehouse Technologies: Production experience with modern lakehouse table formats, ideally Apache Iceberg, including partitioning strategies, table maintenance, compaction, and snapshot management. Experience with Delta Lake or Apache Hudi is also relevant.

Cloud Infrastructure: Experience operating Spark workloads on AWS EMR, Amazon EKS, or comparable platforms, with a strong understanding of cluster management, capacity planning, autoscaling, resource allocation, and infrastructure cost optimization.

Streaming and CDC: Solid understanding of Change Data Capture principles and Kafka semantics, including offsets, consumer configurations, message delivery guarantees, and event processing reliability.

Programming Skills: Strong proficiency in Python and SQL. Experience with Scala or Java is considered an advantage.

Orchestration and Automation: Experience with Apache Airflow or similar workflow orchestration tools, alongside CI/CD pipelines and infrastructure-as-code practices.

Troubleshooting and Analytical Skills: A systematic approach to debugging, with the ability to interpret logs, investigate complex issues, formulate hypotheses, and validate solutions using data.

Communication and Collaboration: Ability to work independently while collaborating effectively with distributed engineering teams and cross-functional stakeholders.

Nice to Have: Experience with Snowflake, particularly querying external or Iceberg tables, as well as monitoring and observability tools such as Grafana, Loki, or Alloy.

Platform Enablement: Previous experience building self-service data platforms or enabling other engineering teams to independently manage data workflows is an advantage.

Containerization: Familiarity with Kubernetes (K8s) is a plus.

What We Offer

~2 min read

Competitive Compensation: Annual gross base salary ranging from $50,000 to $80,000 USD, depending on experience, technical skills, and location.

Localized Compensation: Salary offers are adjusted to the candidate's country and paid in the relevant local currency using market-aligned conversion rates.

Equity Opportunities: Stock grant opportunities, depending on role, employment status, and location.

Flexible Working Arrangements: Fully remote work, with optional flexible office memberships through IWG.

Additional Benefits: Access to additional perks and benefits based on employment status and country of residence.

International Exposure: Opportunity to collaborate with globally distributed engineering teams and contribute to a large-scale international technology platform.

Technical Growth: Work with modern data engineering technologies, including Spark Structured Streaming, Apache Iceberg, AWS, CDC, and AI-assisted operational tools.

Meaningful Technical Ownership: Take responsibility for the architecture, reliability, scalability, and continuous improvement of a business-critical data platform.

Inclusive Workplace: Join an environment committed to equal opportunities, diversity, accessibility, and professional development.

How Jobgether works:

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1

✓Competitive Compensation: Annual gross base salary ranging from $50,000 to $80,000 USD, depending on experience, technical skills, and location.

✓Localized Compensation: Salary offers are adjusted to the candidate's country and paid in the relevant local currency using market-aligned conversion rates.

✓Equity Opportunities: Stock grant opportunities, depending on role, employment status, and location.

✓Flexible Working Arrangements: Fully remote work, with optional flexible office memberships through IWG.

✓Additional Benefits: Access to additional perks and benefits based on employment status and country of residence.

✓International Exposure: Opportunity to collaborate with globally distributed engineering teams and contribute to a large-scale international technology platform.

✓Technical Growth: Work with modern data engineering technologies, including Spark Structured Streaming, Apache Iceberg, AWS, CDC, and AI-assisted operational tools.

✓Meaningful Technical Ownership: Take responsibility for the architecture, reliability, scalability, and continuous improvement of a business-critical data platform.

✓Inclusive Workplace: Join an environment committed to equal opportunities, diversity, accessibility, and professional development.

Location & Eligibility

Where is the job

Brazil

Remote within one country

Who can apply

About this job

Pay per year: $50,000–$80,000 a year; 59% below the $157,500 median for Data Engineer jobs on Jobera (719 salaries).
Who can apply: Brazil
Posted: Today
Employer: has 5,889 open jobs on Jobera in 42 countries.