Senior Data Engineer - Full Stack (US, Remote)

India·PuneRemotesenior

Data EngineerData

0 views0 saves0 applied

Apply Now

Quick Summary

Key Responsibilities

Pipeline execution and orchestration Data governance workflows Metadata synchronization Environment setup and configuration Test harness development Automate workflows on Databricks,

Technical Tools

Data EngineerData

Job Title: Senior Data Engineer - Full Stack
Experience: 8+Years

Role Summary

We are seeking a highly skilled Senior Data Engineer – Full Stack to build and maintain internal tools, automation frameworks, and workflows that enhance the efficiency, reliability, and scalability of our data and machine learning platforms. This role will work closely with Data Engineers, Data Scientists, and ML Engineers to streamline operations across the data lifecycle.

Key Responsibilities

Design and develop CLI tools, scripts, and internal utilities to automate repetitive tasks across the data platform, including:
- Pipeline execution and orchestration
- Data governance workflows
- Metadata synchronization
- Environment setup and configuration
- Test harness development
Automate workflows on Databricks, including:
- Job deployment and scheduling
- Environment provisioning
- MLOps processes using APIs, Terraform, or Databricks SDK
Build and implement robust testing frameworks:
- Integration testing for pipelines
- End-to-end validation of ETL/ELT workflows
- Testing and validation for ML inference workflows
Improve overall productivity, scalability, and reliability of the data and ML engineering ecosystem
Develop lightweight internal tools and dashboards using frameworks such as React, Streamlit, or similar technologies to:
- Visualize data pipelines and workflows
- Demonstrate model inference capabilities
- Provide configuration and operational controls
- Enable internal productivity monitoring and dashboards
Collaborate with cross-functional teams to identify automation opportunities and implement best practices

Required Skills & Qualifications

Strong experience in Python and scripting for automation and backend development
Hands-on experience with Databricks platform and ecosystem
Experience with APIs, Terraform, and/or Databricks SDK for automation
Solid understanding of ETL/ELT pipelines and data platform architecture
Experience building testing frameworks for data pipelines and ML workflows
Familiarity with CLI tool development and system automation
Knowledge of MLOps principles and practices
Experience with modern development practices, including:
- Spec-driven development
- Use of coding agents or automation-assisted development tools
- Version control and CI/CD pipelines

Nice to Have

Experience building dashboards or internal tools using React, Streamlit, or similar frameworks
Familiarity with Databricks AI/BI or other data visualization tools
Exposure to data governance and metadata management frameworks
Experience working with cloud platforms (AWS preferred)

Preferred Experience

8+ years of experience in Data Engineering, Platform Engineering, or related roles
Experience working in data-driven or ML-focused environments

What You’ll Bring

Strong problem-solving mindset with a focus on automation and efficiency
Ability to work in a fast-paced, collaborative environment
Passion for building scalable internal tools and improving developer productivity