Data Engineer with Kafka

“Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors. We are recognized for our deep transformation execution and delivery.

Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial services.

Capco has a tolerant, open culture that values diversity, inclusivity, and creativity.

With no forced hierarchy at Capco, everyone has the opportunity to grow as we grow, taking their career into their own hands.

We believe that diversity of people and perspective gives us a competitive advantage.

• →Strong understanding of Kafka architecture, event streaming concepts, and implementation.
• →Hands-on experience developing and supporting Kafka-based data ingestion and streaming pipelines.
• →Knowledge of Kafka producers, consumers, topics, partitions, consumer groups, and offset management.
• →Understanding of fault tolerance, message delivery guarantees, and streaming performance optimization.

• Strong hands-on experience with Apache Spark for large-scale data processing.
• Expertise in Spark performance tuning, including joins, partitioning, caching, data skew handling, and troubleshooting.
• Experience developing and optimizing production-grade Spark applications.

• Strong proficiency in at least one of Scala or PySpark, with working knowledge of the other.
• Ability to write efficient, maintainable, and production-ready code.

• Advanced SQL skills, including complex joins, window functions, query optimization, and data modeling.
• Experience working with large datasets and optimizing SQL performance.

• Strong experience in at least one orchestration tool, with working knowledge of the others.
• Ability to independently design, develop, monitor, and troubleshoot data pipelines.

• Experience working with cloud-based data lakes and storage solutions.
• Understanding of file formats, partitioning strategies, storage design, data lifecycle management, and performance considerations.

• Good Python scripting and automation skills.
• Experience building utilities, ETL components, and automated testing frameworks.

• Experience handling production incidents, monitoring data pipelines, and performing root cause analysis.
• Ability to implement resiliency improvements and ensure pipeline stability.

• Ability to independently design and deliver medium-to-complex data engineering pipelines with minimal guidance.
• Strong analytical, debugging, and problem-solving skills.
• Good understanding of data engineering best practices, performance optimization, and scalable architecture.
• Experience conducting code reviews and mentoring junior engineers.
• Strong communication and collaboration skills.

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.
• Minimum 5+ years of relevant experience in data engineering, big data technologies, and data pipeline development.
• Hands-on experience with Kafka-based streaming solutions and distributed data processing frameworks.

We are seeking a technically strong, hands-on Senior Data Engineer who can independently own data engineering deliverables, build robust Kafka-based streaming pipelines, optimize large-scale Spark workloads, resolve complex production issues, and contribute to technical design decisions.