Data Engineer - Automation & AI

• →

  Design, build, and evolve AWS Glue–based data pipelines using Spark and SQL. 

• →

  Translate legacy SQL scripts and stored procedures into AWS Glue pipelines 

• →

  Ensure migrated and newly built pipelines meet agreed standards for correctness, performance, and maintainability. 

• Apply Generative AI and agent-based techniques to accelerate data engineering tasks, including code generation/refactoring, pipeline dev. and standardisation 

• Own the design and implementation of AI-assisted tooling that integrates directly into day-to-day engineering workflows 

• Codify successful patterns, reusable tools, and recommended ways of working for scaling beyond the PoC 

• Work hands-on with Python and LLM APIs to build pragmatic, internal DE tools 

• Design effective prompts & interaction patterns for code generation & transformation 

• Evaluate and work with enterprise-grade AI platforms (e.g. AWS Bedrock, Azure AI Foundry) using GPT-4 / Claude-class models 

• Define practical rules of thumb and guardrails (e.g. where automation works, where it breaks down, where human intervention is required). 

• Work closely with data and platform engineers to (dis)prove automation hypotheses and identify where AI adds real productivity gains vs. noise 

• Document outcomes and recommendations from the PoC and provide clear guidance on how AI should (and should not) be used in data engineering at scale 

• 6+ years of experience in data engineering or closely related engineering roles. 

• Proven experience owning and shaping data engineering solutions, not only implementing individual pipelines. 

• Strong hands-on experience with AWS-based data engineering, including: 

  • AWS Glue (jobs, transformations, orchestration) 

  • Spark (batch processing and transformations) 

  • Advanced SQL (complex logic, optimisation, performance tuning) 

  • End-to-end pipeline and workflow design 

• Solid (Python) engineering experience, including building reusable components and internal tooling. 

• Demonstrated, practical experience applying Generative AI in engineering workflows, such as: 

  • Working with LLM APIs (e.g. AWS Bedrock, Azure AI Foundry, OpenAI) 

  • Prompt design for code generation, refactoring, and transformation 

  • Understanding the limitations, failure modes, and risks of LLM-based automation 

• Experience designing AI-assisted engineering workflows or tools, for example: 

  • API-based services (e.g. FastAPI) 

  • MCP (or agent)-like orchestration patterns 

• Able to balance short-term PoC delivery with longer-term capability building. 

• Experience in financial services or other regulated environments is a strong advantage. 

• Ability to be based in the UAE for a minimum of 3 months, working full-time on-site (Abu Dhabi or Dubai). 

This is a unique opportunity to directly impact the data engineering capability of a major financial institution, with real, hands-on use of GenAI - while working on-site in the UAE, at the heart of one of the world’s fastest-growing technology and financial hubs. You will operate in a high-impact setting alongside senior stakeholders on a greenfield AI for Data Engineering initiative with strong executive backing. If you are interested, we’d love to connect. 

Apply now!