Lead Data Engineer

Our client is one of the fastest-growing companies, a leader in digital transformation, Blockchain, and AI technologies with global clients from the education, healthcare, and manufacturing industries. With aggressive plans to expand into many international markets, they are looking to grow their team!

• →Design and deliver cloud-native data solutions on AWS across ingestion, storage, processing, orchestration, serving, and monitoring layers.
• →Design, develop, and maintain robust ETL/ELT pipelines for multi-layer data ingestion.
• →Create and manage scalable feature stores tailored for healthcare analytics.
• →Collaborate closely with ML and analytics teams to operationalise ML workflows.
• →Support production data platforms in terms of performance, reliability, and scalability.

• 5+ years of proven experience in data engineering, building complex, production-grade data pipelines and architecture.
• Extensive experience with the Python data ecosystem, specifically Pandas and NumPy, for large-scale data processing and transformation.
• Strong proficiency in SQL and relational database management, including schema design and query optimization.
• Hands-on experience with AWS data and compute services, including S3, Lambda, and Batch, for building scalable, serverless data pipelines.
• Proficient with boto3 and AWS Wrangler (awswrangler) for programmatic interaction with AWS services and S3-based data lakes, including partitioned data structures.
• Proficiency in Bash scripting and navigating Linux environments for automation and operational tooling.
• Skilled in code optimization, including profiling, memory-efficient processing, and refactoring for performance in data workloads.
• A clear understanding of Data Quality, Data Privacy, and Data Governance principles.
• Working understanding of MLOps practices, including model deployment, monitoring, versioning, and integrating ML inference into production data pipelines.

• Previous experience with IoT Data
• Familiarity with privacy-preserving architecture and data compliance standards.