Data Engineer / Architect III USC and GC - W2 Role

Seeking a highly skilled Data Engineer / Architect III to support enterprise-scale data integration and platform optimization initiatives focused on the Cloudera ecosystem and Apache NiFi. This role is responsible for evaluating, optimizing, modernizing, and operationalizing existing NiFi-based data integration platforms while serving as both a hands-on engineer and a technical advisor.

The ideal candidate will possess strong experience with Apache NiFi, ETL/ELT pipelines, Python automation, Linux environments, and distributed systems.

• Assess and optimize existing Apache NiFi and Cloudera environments.

• Review:

  • Processor Configurations

  • Queue Management

  • Flow Prioritization

  • Back Pressure Settings

  • Clustering Strategies

• Improve platform scalability, reliability, and operational efficiency.

• Establish reusable design standards and governance practices.

• Design, develop, optimize, and support enterprise ETL/ELT pipelines.

• Integrate data from:

  • Relational Databases

  • APIs

  • Streaming Platforms

  • Cloud Storage

  • Distributed File Systems

• Support batch and real-time data ingestion.

• Resolve:

  • Pipeline Failures

  • Data Latency Issues

  • Throughput Bottlenecks

  • Operational Instability

• Develop Python-based automation and transformation scripts.

• Build integrations between NiFi and external systems.

• Automate deployment, validation, monitoring, and operational support processes.

• Support custom orchestration and processing logic.

• Implement enterprise security standards including:

  • Authentication

  • Authorization

  • Encryption

  • RBAC Controls

  • Secure Data Movement

• Support deployment procedures and operational runbooks.

• Assist with version control and CI/CD processes.

• Serve as a technical advisor for platform modernization initiatives.

• Document:

  • Architecture Recommendations

  • Operational Procedures

  • Knowledge Transfer Materials

  • Optimization Findings

• Collaborate with:

  • Data Engineers

  • Architects

  • DevOps Teams

  • Infrastructure Teams

  • Analytics Teams

  • Application Teams

• Apache NiFi

• ETL / ELT Pipelines

• Data Integration Platforms

• Python

• Linux

• Distributed Systems

• Kafka

• Hive

• HDFS

• Iceberg

• CI/CD Pipelines

• Cloud & Hybrid Environments

• Improve performance and reliability of enterprise NiFi environments.

• Optimize large-scale ETL/ELT pipelines.

• Reduce operational overhead through automation.

• Establish reusable platform standards and governance practices.

• Deliver scalable and maintainable data integration solutions.

• Support long-term platform modernization initiatives.

The ideal candidate will:

• Have strong hands-on Apache NiFi experience in enterprise production environments.

• Be highly skilled in ETL/ELT architecture and optimization.

• Possess advanced Python scripting expertise.

• Have strong Linux and distributed systems knowledge.

• Demonstrate excellent troubleshooting and root cause analysis capabilities.

• Be comfortable working independently and advising technical teams.

• Balance architecture strategy with operational execution.

• Apache NiFi (Production Environment)

• Cloudera Ecosystem

• ETL / ELT Development

• Python Scripting & Automation

• Linux Administration

• Distributed Systems

• Data Pipeline Optimization

• Data Integration

• Monitoring & Operational Support

• Root Cause Analysis

• Secure Data Movement & Governance
  

• Apache Kafka

• Hive

• HDFS

• Iceberg

• CI/CD Pipelines

• Cloud & Hybrid Architectures

• Enterprise Data Governance

• Reusable Integration Frameworks

• Distributed Systems Performance Tuning

✔ W2 Only

✔ Green Card Holders Only

✔ Hybrid Role – Cincinnati, OH

✔ Strong Apache NiFi Experience Required

✔ Python & Linux Expertise Required

✔ Data Engineering / Platform Optimization Focus