Senior Solution Architect- Spark & Databricks Expert

United StatesRemoteFull-timesenior

ArchitectConstruction & Real Estate

0 views0 saves0 applied

Apply Now

Quick Summary

Overview

This position is listed on behalf of a partner company, which manages all applications and next steps.

Technical Tools

ArchitectConstruction & Real Estate

Join a technology-focused environment where advanced data engineering and modern analytics solutions help organizations unlock the full value of their data. As a Senior Solution Architect, you will lead complex customer engagements, combining deep technical expertise with strategic consulting and solution delivery. You will design, implement, and optimize large-scale Apache Spark and Databricks workloads, ensuring performance, scalability, and cost efficiency. Acting as a trusted technical advisor, you will independently drive projects from initial discovery through implementation and delivery. You will also collaborate with customers and internal engineering teams to develop innovative solutions and improve data platform capabilities. This role offers the opportunity to influence enterprise data strategies while mentoring technical talent and promoting modern Lakehouse best practices.

Lead customer engagements from initial discovery and technical assessment through solution architecture, implementation, and final delivery, taking full ownership of outcomes.
Design, architect, implement, and optimize large-scale Apache Spark and Databricks workloads, ensuring scalability, reliability, and cost efficiency.
Apply advanced knowledge of Spark internals to troubleshoot, debug, and optimize distributed data processing jobs and SQL workloads.
Develop and deliver proofs of concept (PoCs), technical demonstrations, and customer workshops to communicate solution capabilities and validate architectural approaches.
Act as a trusted technical advisor, proactively identifying customer challenges, anticipating potential issues, and recommending innovative, data-driven solutions.
Drive technical decision-making and translate complex business requirements into practical, scalable data architectures.
Collaborate with product and engineering teams to enhance platform capabilities, improve technical solutions, and strengthen the overall customer experience.
Establish and promote engineering best practices across PySpark, Databricks, and Lakehouse architecture.
Mentor junior engineers and support the development of technical expertise across data engineering teams.
Continuously identify opportunities to improve data processing performance, platform governance, and overall solution efficiency.

Requirements

~1 min read

Between 5 and 10+ years of professional experience in data engineering, analytics, or enterprise data solution delivery.
At least 3 years of hands-on experience with Databricks, including workload optimization, debugging, performance tuning, and governance.
Expert-level understanding of Apache Spark fundamentals and internals, with strong practical experience in PySpark development.
Proven expertise in Databricks SQL (DBSQL), including query optimization and performance tuning.
Strong knowledge of modern Lakehouse architectures, Delta Lake, Structured Streaming, and cloud infrastructure.
Demonstrated ability to architect and deliver complex, large-scale data solutions independently, with minimal supervision.
Strong analytical and problem-solving skills, with the ability to take initiative, make technical decisions, and manage complex challenges autonomously.
Excellent communication and stakeholder management skills, particularly the ability to explain sophisticated technical concepts and architectural decisions to business leaders and non-technical audiences.
Ability to lead customer-facing technical engagements and build trusted relationships through proactive collaboration and solution-oriented thinking.
Preferred: Databricks Certified Data Engineer Professional certification.
Experience with advanced Databricks capabilities, including Delta Live Tables (DLT) and Unity Catalog (UC).
Previous consulting experience involving large-scale data migrations, enterprise analytics transformations, or AI-assisted analytics initiatives is considered an advantage.