P1 - GP - Senior Data Engineer / Data Architect
Quick Summary
build incremental loads from the source platform into Bronze using change timestamps and watermarks, with safe handling of late-arriving rows and full retention of raw history.
Job Description: Senior Data Engineer / Data Architect
About the role
We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks.
The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system. You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications. You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation.
What you'll do
- Ingestion: build incremental loads from the source platform into Bronze using change timestamps and watermarks, with safe handling of late-arriving rows and full retention of raw history.
- Silver: model source data close to its native structure, with deterministic deduplication, type-aware unit conversion, sign rules and quarantine of suspect rows rather than silent fixes.
- Gold: implement a source-agnostic business model of dimensions, fact tables with different history rules and reference data, built as performant Delta tables for Power BI and application use.
- Migration: support a single cutover from the legacy system, and load and freeze the history of companies that will not move.
- Data modeling: work with the data model owner on the item catalog, dimensions (period, currency, fund and entity), cash transactions and derived measures such as multiples and FX conversion.
- Data quality and reconciliation: build checks and quality flags, reconcile new figures against the legacy system before the cutover and run regression checks when the model changes.
- Serving: publish Gold to downstream consumers, including Lakebase (Postgres) synced tables for applications and direct Delta access for BI.
- Operations: orchestrate and monitor the pipelines, manage access in Unity Catalog, and keep documentation, lineage and runbooks current.
- Collaboration: work closely with the vendor's data team, reporting and application teams, and finance and portfolio operations users on definitions and issues.
What you'll bring
- 5+ years building production data pipelines, including 2+ years on Databricks with Delta Lake.
- Strong SQL (Spark SQL and T-SQL) and Python or PySpark.
- Hands-on experience with a medallion (Bronze, Silver, Gold) architecture and incremental processing: watermarks, MERGE, change data feed.
- Dimensional modeling: facts at a clear grain, dimensions, surrogate keys, slowly changing dimensions.
- A track record of data quality work: deduplication, reconciliation, and tracing a wrong number back to its source.
- Experience with Unity Catalog or a comparable governance layer: permissions, lineage, documentation.
- Performance tuning for BI workloads, such as file layout, clustering and query plans.
- Clear written communication, including explaining data issues and trade-offs to non-technical users.
Nice to have
- Private equity, asset management or fund administration data: portfolio monitoring, valuations, cash flows, fund and co-invest structures.
- Financial reporting concepts: LTM and YTD periods, fiscal year ends, budgets and forecasts, restatements, currency conversion.
- Databricks Lakebase or PostgreSQL, including serving data to applications.
- Lakeflow (Delta Live Tables) or Databricks Workflows orchestration.
- Azure SQL and migrating SQL Server logic to Databricks.
- Power BI data modeling, or working closely with BI developers.
- Infrastructure as code and CI/CD for data pipelines, for example Databricks Asset Bundles.
Our environment
Area
Tools
Lakehouse
Databricks, Delta Lake, Unity Catalog
Languages
Spark SQL, Python, PySpark, T-SQL
Application serving
Databricks Lakebase (Postgres) synced tables
Legacy and transition
Azure SQL Database
Reporting
Power BI
Cloud
Microsoft Azure
Location & Eligibility
About this job
- Posted
- Today
Listing Details
- Posted
- November 19, 2026
- First seen
- November 19, 2026
- Last seen
- November 19, 2026
Posting Health
- Days active
- 0
- Repost count
- 0
- Trust Level
- 57%
- Scored at
- November 19, 2026
Signal breakdown
Similar Data Architect jobs
View all →Browse Similar Jobs
Stay ahead of the market
Get the latest job openings, salary trends, and hiring insights delivered to your inbox every week.
No spam. Unsubscribe at any time.