PySpark Developer

Key Responsibilities

• Design, develop, and maintain scalable, high-performance data pipelines using PySpark and Python.
• Build robust ETL/ELT workflows to ingest, transform, validate, and load large volumes of data from multiple sources.
• Develop batch and, where required, streaming data processing solutions to support business and analytical requirements.
• Implement reusable data processing frameworks and ensure pipelines are reliable, maintainable, and scalable.
• Integrate data from databases, APIs, cloud storage, and enterprise data platforms.

• Develop complex data transformations using PySpark DataFrames, Spark SQL, and distributed processing techniques.
• Optimize Spark jobs by analyzing execution plans, partitioning strategies, caching, and resource utilization.
• Troubleshoot performance bottlenecks, resolve processing failures, and improve job execution efficiency.
• Work with large datasets while ensuring efficient memory management and distributed computing performance.
• Apply best practices for handling joins, aggregations, data skew, and incremental data processing.

• Write clean, modular, reusable, and well-documented Python code for data engineering applications.
• Develop utilities, validation scripts, and automation solutions to streamline data workflows.
• Implement exception handling, logging, testing, and debugging practices to improve code quality.
• Collaborate with engineering teams to maintain coding standards and version control practices.

• Implement data validation, reconciliation, and monitoring mechanisms across pipelines.
• Identify and resolve data inconsistencies, duplication, and transformation errors.
• Maintain technical documentation, data lineage, and operational procedures.
• Support production deployments, incident resolution, and continuous improvement initiatives.

• Work closely with business stakeholders and technical teams to translate requirements into scalable data solutions.
• Participate in architecture discussions, code reviews, technical design sessions, and development planning.
• Mentor junior developers and contribute to engineering best practices.

Required Skills and Qualifications

• 8–13 years of overall experience in software development, data engineering, or related technical roles.
• Strong hands-on expertise in PySpark, Apache Spark, and Python.
• Proven experience designing and maintaining scalable data pipelines and ETL/ELT processes.
• Strong understanding of distributed computing, data partitioning, transformations, and performance optimization.
• Experience working with SQL, relational databases, and large-scale structured or semi-structured datasets.
• Familiarity with data warehousing concepts, data modeling, and data quality practices.
• Experience with Git, debugging, testing, and production support.

Preferred Qualifications

• Experience with cloud data platforms such as AWS, Microsoft Azure, or Google Cloud.
• Exposure to platforms such as Databricks, Delta Lake, Snowflake, or cloud-based data lakes.
• Familiarity with orchestration tools such as Apache Airflow and CI/CD practices.
• Knowledge of Kafka or other streaming technologies is an advantage.