Senior Databricks Architect/ Senior Databricks Engineer- Contract- outside ir35

This is a Senior Databricks hands on Architect who is a Databrick engineer by trade. This is a long-term contract engagement to provide dedicated data engineering depth within the Databricks and Shared IEP Services data layer, owning delivery across these workstreams and ensuring the resulting pipelines and schemas are production-grade, governed, and maintainable by the permanent team. The contractor will operate primarily within the Databricks and Shared IEP Services data layer, owning data engineering delivery across three areas:

1. Medallion architecture depth — designing and maintaining Bronze, Silver, and Gold schemas and pipelines using Databricks Delta Lake, including schema evolution and enforcement.

2. Production pipeline engineering — orchestrating and operationally supporting complex pipelines; idempotency, robust error handling, and reconciliation.

3. Unity Catalog governance — fine-grained access control, data lineage, and row/column-level security.

4. Cross-system data integration — handling data cleanliness, integrity, normalization, backfilling, reconciliation, and other ETL challenges across disparate source systems

5. Data privacy & compliance — anonymizing, masking, or tokenizing PII; working understanding of FERPA, COPPA, or GDPR.

6. Performance & cost optimization — Z-ordering, liquid clustering, data skipping, and shuffle-partition management to control cloud spend.

7. Senior engineering experience — 8+ years overall software engineering, including 3+ years in a senior data engineering role.

1.     Familiarity with Databricks AI development capabilities (e.g. vector search, model serving, LLM pipelines).

2.     EdTech engineering experience.

3.     Familiarity with EdTech specifications and standards such as 1EdTech, EdFi, CEDS, or OneRoster.