2h ago
New

Lead Data Engineer, Databricks

United StatesUnited StatesRemoteFull-timelead
Data EngineerData
2 views0 saves0 applied

Quick Summary

Key Responsibilities

Lead data engineering activities across customer engagements, from initial design and implementation through production handover. Build, maintain,

Technical Tools
Data EngineerData

The Lead Data Engineer will drive the delivery of production-grade data engineering solutions for enterprise customers, taking ownership of technical workstreams from design through production handover. This role combines hands-on development with technical leadership, building scalable batch and streaming pipelines using Databricks, PySpark, SQL, and Delta Lake. You will establish engineering standards, guide implementation decisions, and ensure solutions meet performance, quality, and cost objectives. Working directly with customers and cross-functional technical teams, you will translate complex requirements into reliable data platforms. You will also mentor engineers, collaborate with data architects, and communicate technical trade-offs to stakeholders. This is a fully remote opportunity for an experienced engineer who enjoys combining technical excellence, customer engagement, and team leadership.

The Lead Data Engineer will serve as the technical owner of engineering workstreams across customer engagements, ensuring that data solutions are designed, implemented, optimized, and delivered to production. Key responsibilities include:

  • Lead data engineering activities across customer engagements, from initial design and implementation through production handover.

  • Build, maintain, and optimize batch and streaming data pipelines using PySpark, SQL, and Delta Lake on Databricks.

  • Implement Medallion architecture across cloud environments and establish consistent data layer conventions and engineering patterns.

  • Take ownership of platform performance and cost efficiency by benchmarking Spark jobs, tuning SQL queries, right-sizing compute resources, and evaluating technical trade-offs.

  • Implement data quality and validation processes as code, including Lakeflow expectations and automated testing within CI pipelines.

  • Establish and maintain engineering standards, ensuring the reliability, scalability, and quality of delivered solutions.

  • Mentor junior and mid-level engineers through code reviews, collaborative development, technical guidance, and design feedback.

  • Partner with Data Architects to validate architectural decisions and identify potential design limitations during implementation.

  • Communicate technical decisions, delivery progress, and implementation trade-offs clearly to customers and non-technical stakeholders.

  • Work closely with customer teams and relevant platform specialists to ensure successful delivery of enterprise data solutions.

  • Requirements:

    The ideal candidate is a highly experienced data engineer with a strong background in Databricks and cloud-based data platforms, combined with proven technical leadership and customer-facing communication skills. Required qualifications include:

  • At least 7 years of professional data engineering experience, including experience leading projects or engineering teams.

  • Strong proficiency in Python, SQL, Apache Spark, and Delta Lake.

  • Hands-on production experience building both batch and streaming data pipelines at enterprise scale.

  • Practical experience with at least one major cloud platform: AWS, Microsoft Azure, or Google Cloud Platform (GCP), including its data services.

  • Experience with data modeling, Git, and CI/CD practices for data engineering workloads.

  • Demonstrated ability to optimize Spark performance, improve pipeline efficiency, and manage platform costs.

  • Strong understanding of scalable data architecture and modern data engineering practices.

  • Excellent client communication skills, including the ability to explain complex technical concepts and trade-offs to non-technical stakeholders.

  • Proven ability to guide engineers, provide constructive feedback, and take ownership of technical delivery.

  • Nice to Have:

  • Experience in healthcare or life sciences environments.

  • Databricks Certified Data Engineer Associate or Professional certification.

  • Hands-on experience with Unity Catalog, Lakeflow, or Databricks Asset Bundles.

  • Familiarity with dbt, Airflow, or equivalent orchestration and data transformation tools.

  • Experience migrating legacy ETL systems to modern cloud-based data platforms.

  • Previous consulting or client-facing engineering experience.

  • Benefits:

  • Unlimited paid time off.

  • Medical, dental, and vision coverage.

  • Retirement savings plan, including 401(k) and IRA options.

  • Basic and voluntary life insurance.

  • Accidental Death and Dismemberment (AD&D) coverage.

  • Maternity and paternity leave.

  • Short-term and long-term disability coverage.

  • Training and professional development opportunities.

  • Dedicated certification roadmap for technical employees.

  • In-house training delivered by Databricks-approved trainers.

  • Dedicated Employee Success Manager focused on supporting the employee experience.

  • How Jobgether works:
    We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
    We appreciate your interest and wish you the best!
     
    Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
     
     
    #LI-CL1

    Location & Eligibility

    Where is the job
    United States
    Remote within one country
    Who can apply
    US

    About this job

    Who can apply
    United States
    Posted
    Today
    Employer
    has 4,414 open jobs on Jobera in 42 countries.

    Listing Details

    Posted
    November 16, 2026
    First seen
    November 16, 2026
    Last seen
    November 16, 2026

    Posting Health

    Days active
    0
    Repost count
    0
    Trust Level
    68%
    Scored at
    November 16, 2026

    Signal breakdown

    freshnesssource trustcontent trustemployer trust
    Newsletter

    Stay ahead of the market

    Get the latest job openings, salary trends, and hiring insights delivered to your inbox every week.

    A
    B
    C
    D
    Join 12,000+ marketers

    No spam. Unsubscribe at any time.

    Lead Data Engineer, Databricks