Senior Databricks Architect & Apache Spark Engineer - Senior Vice President

We are looking for a highly experienced Senior Databricks Architect & Lead Engineer to provide architectural and technical leadership for core data processing platform on Databricks on AWS. While leading the transition from the legacy Cloudera Hadoop platform to Databricks on AWS, this role owns the long‑term evolution, optimization, and modernization of the Databricks platform The ideal candidate combines deep Databricks and Spark expertise, strong architectural vision, and hands‑on leadership to drive complex transformations and sustained platform excellence at scale. Responsibilities: Architecture Ownership * Define and evolve the target‑state Databricks architecture, ensuring alignment with enterprise and cloud standards Migration & Platform Evolution Leadership * Lead migration roadmap, sequencing, and refactoring strategy * Drive continuous modernization, simplification, and performance improvement beyond initial migration Technical Leadership & Governance * Serve as the primary technical authority for the Databricks platform * Establish and enforce engineering standards, design reviews, quality gates, and architectural compliance * Mentor engineers and lead critical design and decision forums Advanced Databricks & Spark Architecture * Drive adoption of advanced Databricks capabilities * Define and own the performance, scalability, and cost‑optimization framework Platform Modernization & Innovation * Simplify legacy pipelines and introduce reusable frameworks and accelerators * Improve observability and operational resilience * Enable new capabilities such as data sharing, real‑time analytics, and AI/ML integration, ensuring a future‑ready platform Collaboration & Stakeholder Engagement * Partner with enterprise architects and platform teams * Act as single point of accountability for technical delivery * Communicate architecture, risks, and roadmap progress to senior stakeholders Qualifications: * 15+ years of relevant experience * Core Technical Expertise: Deep expertise in Apache Spark (internals, optimization, execution models) and Databricks on AWS * Architecture & Modernization Experience: Proven track record leading large‑scale platform modernization and cloud data transformations * Leadership & Governance: Experience leading engineering teams and architecture review forums. Ability to define standards, enforce them, and influence across teams and stakeholders * Performance & Scalability: Advanced Spark tuning at scale and distributed system bottleneck analysis, Experience designing high‑throughput, low‑latency data pipelines * Behavioral Competencies: Strategic thinker with strong execution focus, Ability to navigate ambiguity and drive clarity, Strong leadership, mentoring, and communication skills Education: * Bachelor’s degree/University degree or equivalent experience \------------------------------------------------------ ## Job Family Group: Technology \------------------------------------------------------ ## Job Family: Applications Development \------------------------------------------------------ ## Time Type: Full time \------------------------------------------------------ ## Most Relevant Skills Please see the requirements listed above. \------------------------------------------------------ ## Other Relevant Skills For complementary skills, please see above and/or contact the recruiter. \------------------------------------------------------ Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law. If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi. View Citi’s EEO Policy Statement and the Know Your Rights poster.