Data bricks Lead Data Engineer - R01565550

Brillio is a leading digital transformation partner delivering innovative solutions across Data, Cloud, AI, and Digital Engineering. We help global enterprises unlock value through cutting-edge technology and data-driven insights.

We are looking for a hands-on Data Engineer with strong Databricks expertise to design, build, and scale modern data platforms. The ideal candidate will have experience in PySpark, Delta Lake, and cloud-native data engineering and will work closely with analytics, product, and business teams to deliver high-performance data pipelines.

• →Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark
• →Build and optimize data lakehouse architectures (Bronze–Silver–Gold layers)
• →Develop data ingestion frameworks from multiple sources (batch & real-time)
• →Implement Delta Lake features including schema evolution, ACID transactions, and optimization
• →Create and manage Databricks notebooks, workflows, and jobs for production pipelines
• →Ensure data quality, governance, and lineage across pipelines
• →Optimize Spark jobs for performance (partitioning, caching, tuning)
• →Collaborate with cross-functional teams (Data Scientists, Analysts, Architects)
• →Support production deployments, monitoring, and troubleshooting of data pipelines
• →Contribute to migration of legacy data platforms to modern cloud-based architecture

---

• Strong experience in:
• Databricks (Workflows, Notebooks, Jobs)
• PySpark / Spark SQL
• SQL & Data Warehousing concepts
• Hands-on experience with:
• Delta Lake / Lakehouse architecture
• Data pipeline development & orchestration
• Good understanding of:
• Data modeling, ETL/ELT processes, SCD concepts 

---

• Experience in at least one:
• AWS (S3, Glue, EMR) / Azure (ADF, ADLS, Synapse)
• Workflow tools:
• Airflow / Databricks Workflows
• Version control:
• Git / CI-CD pipelines

---

• Experience with:
• Streaming frameworks (Kafka / Spark Streaming)
• DBT / Snowflake / Redshift / Fabric
• Exposure to:
• Unity Catalog, Autoloader, Delta Live Tables 

---

• 5–10 years of experience in Data Engineering
• Minimum 2–4 years of hands-on Databricks experience
• Proven experience building enterprise-scale data platforms and pipelines

---

• Strong problem-solving and analytical skills
• Ability to work in a fast-paced, agile environment
• Good communication and stakeholder management

---