Senior Data Engineer

India·BangaloreFull-timesenior

Data EngineerData

0 views0 saves0 applied

Technical Tools

Data EngineerData

~1 min read

~1 min read

→ Design, develop, and maintain scalable data pipelines, ETL/ELT workflows, and data processing systems.
→ Perform advanced data mining across large and complex datasets to identify trends, relationships, anomalies, and valuable insights.
→ Analyze datasets to identify meaningful patterns and correlations that can support business and product decisions.
→ Build and optimize data architectures capable of handling large-scale data volumes and high-throughput workloads.
→ Develop reliable batch and real-time data processing solutions using distributed computing technologies.
→ Work with structured and unstructured data from multiple sources and ensure data is accurately processed and transformed.
→ Optimize data pipelines, queries, storage, and processing frameworks for performance, scalability, and reliability.
→ Implement data quality, validation, monitoring, and error-handling mechanisms across data workflows.
→ Collaborate with Data Scientists, Analysts, Software Engineers, Product Managers, and other stakeholders to understand data requirements.
→ Design and maintain data models, schemas, and data integration frameworks.
→ Troubleshoot complex data and pipeline issues and drive improvements in system reliability and efficiency.
→ Contribute to technical architecture decisions and establish best practices for large-scale data engineering.
→ Mentor junior and mid-level engineers and contribute to improving engineering standards and processes.

Data Mining
Strong understanding of pattern identification, trend analysis, and data exploration
Experience working with Large Scale Systems
Strong knowledge of data structures, algorithms, and database concepts
Hands-on experience building ETL/ELT pipelines and data processing workflows
Experience with SQL and relational databases
Proficiency in at least one programming language such as Python, Java, or Scala
Experience with distributed data processing and large-scale data platforms
Strong understanding of data modeling, data warehousing, and data architecture
Experience optimizing data pipelines and processing workloads for performance and scalability
Strong analytical and problem-solving abilities

Experience with Apache Spark, Kafka, Hadoop, Flink, or similar distributed technologies
Exposure to cloud platforms such as AWS, Microsoft Azure, or Google Cloud
Experience with data lakes, lakehouses, and modern data warehouse technologies
Knowledge of real-time and streaming data architectures
Experience with data governance, observability, and data quality frameworks
Familiarity with machine learning or analytics workflows is a plus
Experience working with high-volume, high-velocity, or complex datasets