Freelance Senior Databricks Platform Engineer

The role we are currently looking for is closer to a Databricks Platform Engineer than a traditional Data Engineer or BI consultant. The objective is to help design and build a Data Product Platform on Databricks, with a strong focus on platform architecture, governance, Infrastructure as Code, CI/CD, security, and enabling multiple teams to build and operate data products.

• Design and implement Lakehouse architectures on Databricks (medallion / bronze‑silver‑gold patterns, Delta Lake, Unity Catalog).

• Build robust batch and streaming data pipelines using PySpark and Spark SQL, orchestrated via Databricks Workflows / Jobs.

• Model data for analytics use cases (e.g. star schema for BI, curated datasets for data science).

• Ingest data from APIs, files, databases and SaaS sources into data lakes and lakehouses.

• Implement clear and testable transformation logic (data quality checks, deduplication, SCD, CDC, business rules).

• Apply data governance best practices (cataloguing, lineage, RBAC, masking of sensitive data).

• Set up CI/CD and deployment pipelines (Azure DevOps / GitHub) for Databricks assets (notebooks, jobs, pipelines, workflows).

• Monitor, optimize and troubleshoot pipelines and clusters for performance, robustness and cost efficiency.

• Help define and implement best practices, coding standards and reusable components within the team.

• Databricks platform architecture

• Unity Catalog and governance

• CI/CD and DevOps for Databricks - Enterprise GitHub

• Terraform / Infrastructure as Code

• Lakehouse architecture

• Data Product Platform enablement

• The Knowledge Graph, ontology and agentic AI are considered a strong plus.

• Unity catalogue, CICD, IAC and terraform skills in mode implementation and integration