Data Engineer (Databricks)

• Pipeline Development: Design, build, and maintain scalable batch and incremental pipelines in Databricks (PySpark, Spark SQL, Delta Lake) that ingest data from multiple source systems into a medallion (bronze/silver/gold) architecture.

• Data Integration & Modelling: Implement the canonical data model defined with the Data Architect, so figures reconcile consistently across every product surface.

• Entity Resolution: Build production pipelines that resolve duplicate and conflicting records across systems that were never designed to reconcile with one another.

• Data Quality Engineering: Profile source data, implement automated data quality checks and expectations, and build monitoring and reporting so data quality stays continuously visible against defined targets.

• Orchestration & Reliability: Orchestrate and monitor jobs with Databricks Workflows, optimise Spark performance and cost, and troubleshoot pipeline failures.

• Governance & Privacy: Apply Unity Catalog governance, multi-tenant data isolation, and anonymisation and aggregation rules for pooled cross-customer analytics.

• Collaboration: Partner with the Data Scientist to productionise comparability models and data quality metrics, and with Engineering to expose curated data to the product.

• 5+ years in data engineering or analytics engineering, building production data pipelines in a cloud environment.

• Hands-on Databricks experience: PySpark and Spark SQL, Delta Lake, Databricks Workflows/Jobs, and Unity Catalog.

• Strong SQL and Python, with solid data modelling skills (dimensional and/or canonical models) and experience with the lakehouse/medallion architecture.

• Practical experience with deduplication, record linkage, or entity resolution, and with diagnosing and remediating data quality problems at scale.

• Experience implementing automated data quality checks, pipeline monitoring, and Spark performance and cost optimisation.

• Experience with cloud platforms (AWS preferred), Git-based version control, and CI/CD for data pipelines.

• Clear communication with data scientists, architects, and product stakeholders, including flagging data limitations early.

• Working proficiency with AI tools (e.g., Claude, Databricks Assistant, GitHub Copilot) to accelerate pipeline development, debugging, and documentation, with the judgment to validate AI-generated code before production use.

• Databricks certification (e.g., Data Engineer Associate or Professional).

• Experience with streaming or incremental ingestion (e.g., Auto Loader, Structured Streaming, Delta Live Tables).

• Experience building data for benchmarks, scoring models, or analytics intended for external consumption.

• Experience with anonymisation, aggregation thresholds, or privacy-preserving analytics.

• Background in enterprise B2B SaaS or the legal technology domain.