Hadoop Administrator

We are looking for an experienced Hadoop Administrator to manage, maintain, and upgrade our on-premises Hadoop clusters. The role involves working on Apache Hadoop versions 2.7.1 and 3.4.x, ensuring cluster stability, performance, and scalability, along with ecosystem components like Hive and YARN.

- Install, configure, and manage on-prem Hadoop clusters based on Apache Hadoop (versions 2.7.1 and 3.4.x)

- Administer and monitor Hadoop Distributed File System (HDFS) including storage management, replication, and data balancing

- Manage resource allocation and scheduling using Apache YARN

- Hadoop upgrades (2.7 → 3.4) in on-prem environments

- Perform Cluster migration, Data validation (HDFS checksums, block integrity) & Backward compatibility validation for jobs

- Experience with migration tools and distribution changes

- Administer and optimize Apache Hive (query tuning, metastore management)

- Work with ecosystem tools like Spark (basic admin exposure preferred) & Tez (execution engine for Hive)

- Ensure high availability for Hive Metastore and critical services

- Monitor cluster health, job performance, and resource utilization

- Tune YARN queues and capacity scheduler, Hive queries and execution plans, Disk, memory, and network utilization

- Implement performance improvements for large-scale data processing

- Configure authentication/authorization

- Ensure data security and compliance policies

- Manage access controls and audit logs

- Provide support for Hadoop cluster issues

- Troubleshoot Node failures, Job failures (Hive/YARN) & Data inconsistencies

- Perform log analysis and root cause identification

- Plan and manage backup and disaster recovery strategies

- Implement DistCp-based replication & DR cluster setup (active-passive / active-active)

- Strong hands-on experience with Apache Hadoop (2.7.x and 3.x)

- Expertise in Apache YARN, Apache Hive & HDFS administration

- Experience in on-prem cluster setup and hardware-level optimization

- Linux administration (RHEL/CentOS/Ubuntu)

- Shell scripting for automation