Technical Lead Data Engineering

We are now seeking strategic and hands‑on data leader to guide our high‑performing team in delivering smarter, more efficient regulatory outcomes whilst playing a key role to uplift processes, strengthen governance and drive reliable data solutions.  

As Technical Lead, Data Engineering you will bring expertise across data architecture, data lifecycle management, AI software development, information security, and agile delivery.  
 
Key responsibilities of this role are:  

• Be accountable for the quality of data pipeline development and the flow of data on the NHVR Enterprise Data Platform. Quality is measured by data integrity, pipeline performance, uptime, efficient administration, cost, data security, data governance, and architectural integrity.
• Own and evolve the platform architecture and technical roadmap. This includes leading the transition from notebook- and job-based pipelines toward Lakeflow declarative pipelines and other modern Databricks capabilities.
• Own the design and continuous improvement of the metadata-driven ingestion and transformation framework. Ensure it stays scalable, maintainable, well-documented, and consistently applied.
• Define, document, and enforce engineering standards, patterns, and practices, covering coding standards, Delta Lake and medallion design, testing, CI/CD, and observability. Ensure compliance through design reviews and code reviews.
• Provide authoritative advice to Managers, Directors, and other senior staff across the organisation regarding architecture, augmentation, implementation, and manageability of the data on the NHVR Enterprise Data Platform.  
• Provide technical guidance, mentoring, and coaching to Data Engineers, including pairing on complex problems and building team capability in Databricks and modern engineering practice.  
• Lead technical planning within the Scrum team: break down epics, assess solution options, estimate technical effort, and identify technical risks and dependencies early.
• Act as the technical escalation point for complex incidents, leading root cause analyses and driving lasting improvements to monitoring, alerting, and resilience.
• Represent data engineering changes at CAB and ensure releases meet quality, security, and change management requirements.
• Remain hands-on, delivering complex or high-risk pipeline and framework components alongside the team.  

NHVR’s Data Science and Analytics Unit are currently undertaking growth and considerable transformation, and this is an exciting time to join the organisation. 

To learn more, the position description can be found on the NHVR Careers site -https://careers.nhvr.gov.au/jobs/search 

To be successful in this role you will possess a Master’s degree in information technology or computer science related or equivalent.  You will have a minimum of 10 years’ experience with demonstrated skills in Databricks and Azure cloud platform (certifications in either / both would be highly regarded) 

• Expert Python and PySpark development skills, with advanced SQL (Spark SQL and T-SQL or similar).
• Deep Databricks expertise, including notebooks, Jobs/Workflows, cluster configuration and policies on classic compute, and advanced Spark performance tuning.
• Advanced Delta Lake knowledge, including MERGE and upsert patterns, schema evolution, OPTIMIZE and VACUUM, liquid clustering and partitioning strategy, and time travel.
• Demonstrated experience designing, building, or owning metadata-driven or configuration-driven pipeline frameworks.
• Demonstrated experience designing and implementing Unity Catalog security and governance models, including grants and ownership, row and column level security, external locations, storage credentials, and lineage.
• Proven experience establishing approaches for identifying, classifying, and protecting sensitive and PII data.
• Strong data architecture and modelling skills, covering relational, dimensional, and medallion architecture, normalisation, schemas, and data profiling.
• Proven code review, software analysis, and design review skills, with a track record of defining and enforcing engineering standards.
• Strong software engineering discipline, including modular design and design patterns, unit and data quality testing (e.g. pytest, data quality frameworks), Git-based workflows, CI/CD with Azure DevOps, and debugging.
• Extensive experience with ELT/ETL pipelines and data integration, including REST APIs, file-based exchange (e.g. SFTP), change data capture, and streaming or event-based patterns.
• Experience leading technical teams in Scrum and Agile methods, with the ability to mentor engineers and influence without direct authority.
• Excellent communication skills, including explaining complex technical matters and trade-offs to senior, non-technical stakeholders