Senior Databricks Architect / Lead Databricks Engineer

We're seeking an exceptional Senior Databricks Architect with a strong hands-on Data Engineering background to join a major long-term transformation programme.

This is not a pure architecture role. You'll be a Databricks specialist who remains close to the technology, designing and building enterprise-scale data solutions while driving engineering excellence across critical data platforms.

You'll take ownership of delivering robust, scalable, and production-grade data solutions within a modern Databricks ecosystem, ensuring platforms are governed, secure, maintainable, and optimised for long-term growth.

🔹 Key Responsibilities

• Architect, build and optimise enterprise-grade data solutions within Databricks

• Design and implement scalable Delta Lake data architectures following Medallion principles

• Lead data engineering delivery across multiple strategic workstreams

• Develop highly resilient, production-ready pipelines with strong operational controls

• Drive data governance, security, lineage and compliance best practices

• Collaborate with product, engineering and business teams to deliver trusted data assets

• Mentor internal engineering teams and establish engineering standards

🔹 Core Workstreams

📊 Product Data Ingestion

• Design and maintain large-scale ingestion pipelines

• Integrate assessment, content and product data into shared enterprise data platforms

• Ensure data quality, consistency and operational reliability

🤖 AI & GenAI Feature Enablement

• Build and maintain data foundations powering LLM and GenAI capabilities

• Design schemas and pipelines supporting AI-driven products and document processing

• Handle complex parsed, normalised and sensitive datasets at scale

• Support emerging AI use cases including evaluation processing, bulk uploads and intelligent workflows

🏥 IEP & Medicaid Data Integration

• Build robust integrations across IEP, Medicaid and associated service platforms

• Deliver reconciliation, data quality and cross-system consistency

• Support enterprise reporting and operational analytics requirements

🔹 Essential Experience

✅ Deep expertise in Databricks and Delta Lake architecture

✅ Proven experience implementing and operating Medallion Architecture (Bronze, Silver, Gold)

✅ Advanced pipeline engineering experience including:

• Orchestration

• Idempotent processing

• Error handling

• Monitoring

• Recovery strategies

• Data reconciliation

✅ Strong Unity Catalog experience including:

• Data governance

• Lineage

• Fine-grained permissions

• Row-level and column-level security

✅ Extensive ETL and data integration expertise across complex enterprise systems

✅ Strong understanding of:

• Data quality frameworks

• Data normalisation

• Backfilling strategies

• Data integrity controls

✅ Experience handling sensitive and regulated data environments

✅ Knowledge of PII protection techniques including:

• Masking

• Tokenisation

• Anonymisation

✅ Familiarity with GDPR and broader data privacy standards

✅ Performance and cost optimisation expertise including:

• Z-Ordering

• Liquid Clustering

• Data Skipping

• Query tuning

• Shuffle optimisation

• Cloud cost control

✅ 8+ years software engineering experience

✅ 3+ years operating at Senior/Lead Data Engineer or Databricks Architect level

🔹 Highly Desirable

⭐ Databricks AI capabilities

• Vector Search

• Model Serving

• LLM Pipelines

• Retrieval-Augmented Generation (RAG)

⭐ EdTech domain experience

⭐ Experience with industry standards such as:

• Ed-Fi

• OneRoster

• CEDS

• 1EdTech

🔹 Why Apply?

• £700/day Outside IR35

• Fully Remote

• Long-term 12-month engagement with strong extension potential

• High-impact programme combining Data Engineering, AI and Enterprise Architecture

• Opportunity to shape the future data platform of a growing organisation

• Significant ownership and influence from day one