AWS Lead Data Engineer - R01564133

We are looking for a Senior Data Engineer with strong Databricks and Big Data expertise to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have hands-on experience with Spark, PySpark, AWS ecosystem, and modern data architectures, along with exposure to APIs and real-time data processing.

• →Design and build scalable data pipelines using Spark / PySpark on Databricks
• →Develop and optimize ETL/ELT workflows for large-scale data processing
• →Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing
• →Build and maintain data lake and data warehouse solutions
• →Develop and integrate APIs for data ingestion and consumption
• →Implement data processing workflows using Airflow / Autosys
• →Optimize data performance using partitioning, caching, and query tuning
• →Handle structured and semi-structured data from multiple sources
• →Ensure data quality, governance, and reliability of pipelines
• →Collaborate with cross-functional teams (Analytics, Product, Engineering)

---

• Strong expertise in:
• Apache Spark / PySpark
• Databricks (mandatory)
• SQL (PostgreSQL or similar)
• Hands-on experience with:
• AWS EMR, S3
• Hadoop, Hive ecosystem
• Programming skills:
• Python (must-have)
• Scala (good exposure)
• Strong knowledge of:
• UNIX / Shell scripting
• ETL pipelines and data engineering fundamentals

---

• Experience with:
• Elasticsearch (data storage & retrieval)
• Workflow orchestration tools (Airflow, Autosys)
• Exposure to:
• API development & integration
• Version control tools (Git / SVN)
• Basic knowledge of HTML (for web/data tasks)

---

• Minimum 5+ years of Data Engineering experience
• Strong hands-on experience in Databricks + Big Data stack
• Proven experience working on large-scale distributed systems

---

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field

---