Lead Software Engineer - Cloudera Data Platform

• →Administer Cloudera Data Platform services and supporting components, including Cloudera Manager, HDFS, YARN, Hive, Impala, Spark, Ranger, Atlas, Knox, ZooKeeper, Kerberos, Ozone , OTF Iceberg and related platform services where deployed. 

• →Manage service configuration, role placement, parcels and runtime versions, host maintenance, commissioning and decommissioning, rolling restarts, health checks, configuration validation, and environment consistency. 

• →Monitor and troubleshoot cluster health, HDFS and Ozone capacity, replication, under-replicated data, YARN queues, workload contention, Hive Metastore, Impala admission control, Spark execution, service logs, JVM resources, and host-level dependencies. 

• →Administer data-platform security, including Kerberos integration, Ranger policies, Knox access, TLS certificates, keytabs, service identities, encryption-related configuration, and audit logging. 

• →Support backup, restore, replication, disaster recovery, data retention, metadata protection, and recovery testing for platform services and configuration. 

• →Plan cluster expansion, operating system and Java compatibility changes, platform upgrades, service migrations, storage evolution, and decommissioning of obsolete components. 

• →Administer clustered Apache NiFi environments, including nodes, process groups, processors, controller services, parameter contexts, reporting tasks, connection pools, repositories, and registry integration. 

• →Operate and troubleshoot data flows, queues, back pressure, provenance, bulletins, failed transactions, state management, priorities, concurrent tasks, scheduling, record processors, and interactions with source and target systems. 

• →Define standards for reusable flow design, version control, promotion between environments, parameterization, secret handling, naming, documentation, error routing, retry logic, dead-letter patterns, and operational ownership. 

• →Manage NiFi security, including TLS, certificates, authentication, authorization policies, service accounts, proxy configuration, sensitive properties, and integration with enterprise identity services. 

• →Plan and execute NiFi upgrades, node replacement, cluster expansion, repository maintenance, certificate renewal, flow migration, backup and recovery activities, and resilience testing. 

• →Monitor flow throughput, queue growth, back pressure, repository utilization, JVM resources, garbage collection, thread usage, connection status, and end-to-end delivery behavior. 

• →Administer Apache Airflow environments and core components, including schedulers, webservers, triggers, workers, executors, metadata databases, DAG processing, connections, variables, pools, queues, plugins, and providers. 

• →Operate and troubleshoot DAG parsing, task scheduling, retries, dependencies, sensors, deferrable operators, worker capacity, queue behavior, executor performance, metadata database contention, failed tasks, and scheduler availability. 

• →Define standards for DAG structure, deployment, version control, environment promotion, secrets, connections, variables, pools, concurrency, retries, alerting, ownership, documentation, and operational readiness. 

• →Manage Airflow security, including authentication, role-based access control, secrets backends, service identities, TLS, API access, auditability, and integration with enterprise identity and security services. 

• →Plan and execute Airflow upgrades, provider updates, executor or worker changes, metadata database maintenance, DAG migration, backup and restore validation, and high-availability improvements. 

• →Monitor scheduler heartbeat, DAG processing time, task latency, queue depth, worker utilization, executor state, triggerer health, metadata database performance, and availability of dependent services. 

Ensure secure, reliable, and stable operation, availability, performance, and resilience of Cloudera, Apache NiFi, and Apache Airflow services across production and non-production environments, including platform security, access control, authentication, authorization, encryption, and compliance with enterprise security standards.

Define and review high-availability concepts, disaster recovery arrangements, environment topology, service placement, integration patterns, configuration standards, naming conventions, and operational guidelines. 

Analyze and resolve complex issues across distributed services, workloads, data flows, workflows, metadata, security, networking, operating systems, storage, databases, JVMs, and dependent infrastructure. 

Consult teams on solution onboarding, workload design, resource utilization, scheduling, data movement, dependency management, resilience and error handling, operational readiness, and production support requirements.

Maintain authentication, authorization, roles, policies, service accounts, certificates, TLS, secrets, network connectivity, encryption settings, and integrations with enterprise identity and security services. 

Monitor workload concurrency, queues, memory, CPU, storage, network throughput, JVM behavior, NiFi back pressure, repository usage, Airflow scheduler performance, task throughput, metadata database health, and service saturation; recommend scaling and tuning actions. 

Provide maintenance, refresh, patching, configuration alignment, access management, test support, troubleshooting, and lifecycle activities for DEV, TEST, SIT, UAT, PROD, and similar environments. 

Create and improve scripts, configuration templates, deployment automation, health checks, dashboards, alerts, reporting, and self-service procedures using infrastructure-as-code and CI/CD practices. 

Coordinate version currency, upgrades, patches, vulnerability assessment, remediation plans, exception handling, risk acceptance, end-of-support tracking, and compliance evidence for platform components. 

Plan and execute platform upgrades, migrations, configuration changes, certificate renewals, component replacements, backup and restore validation, failover exercises, and disaster recovery tests with controlled business impact. 

Define monitoring standards, alert thresholds, service dashboards, availability views, incident trends, capacity reports, and operational KPIs for the supported services. 

Maintain architecture records, technical documentation, runbooks, standards, configuration inventories, risk records, audit evidence, operational controls, and support procedures. 

Provide expert guidance on distributed data platforms, data ingestion, orchestration, resilience, security, platform selection, and operational best practices. 

Support critical incidents, root cause analysis, post-incident reviews, permanent corrective actions, and knowledge sharing across operations, engineering, architecture, and application teams. 

• Deep knowledge of distributed data platform operations, data ingestion, batch and streaming integration, workflow orchestration, scheduling, dependency management, metadata, and data lifecycle concepts. 

• Hands-on production administration experience with Cloudera Data Platform, Apache NiFi, and Apache Airflow. 

• Strong troubleshooting capability across application, platform, JVM, database, operating system, network, storage, and security layers. 

• Ability to analyze logs, metrics, traces, service states, resource queues, flow files, back pressure, provenance, DAG runs, task instances, scheduler behavior, and infrastructure dependencies. 

• Ability to lead root cause analysis and define permanent corrective actions for recurring platform, integration, workload, or orchestration issues. 

• Experience with high availability, clustering, failover, disaster recovery, backup and restore, business continuity, and recovery testing. 

• Strong understanding of security controls: Kerberos, TLS, certificates, LDAP/Active Directory integration, role-based access control, Apache Ranger, secrets management, encryption, and audit logging. 

• Good scripting and automation skills using Bash, Python, Ansible,  Git, CI/CD tooling, or equivalent technologies. 

• Strong Linux administration skills and practical knowledge of JVM tuning, systemd, networking, DNS, load balancing, firewalls, storage, databases, and container platforms. 

• Knowledge of IT service management processes, including incident, problem, change, release, configuration, vulnerability, lifecycle, and capacity management. 

• Ability to maintain clear technical documentation, runbooks, support models, architecture records, operational procedures, and audit evidence. 

• Strong communication and stakeholder management skills, including the ability to coordinate technical activities during incidents, changes, upgrades, migrations, releases, and service improvements. 

Cloudera Manager, CDP Private Cloud Base, HDFS, Ozone, OTF Iceberg, YARN, Hive, Impala, Spark, Ranger, Atlas, Knox, ZooKeeper, Kerberos, Hive Metastore, resource management, workload governance, backup and disaster recovery. 

NiFi clustering, processors, controller services, process groups, queues, back pressure, FlowFile and content repositories, provenance, state management, parameter contexts, Registry, record-based processing, Site-to-Site, APIs, security, monitoring, and performance tuning. 

Schedulers, webservers, triggerers, workers, DAG processors, executors, metadata database, DAGs, operators, sensors, task instances, pools, queues, connections, variables, providers, plugins, secrets backends, REST API, RBAC, monitoring, and performance tuning. 

Python, Bash, Ansible, Git, CI/CD pipelines, infrastructure as code, REST APIs, configuration management, container images, Kubernetes, Helm, and artifact repositories. 

SQL, JDBC/ODBC, relational databases, object storage, Kafka or comparable messaging services, SFTP, REST services, and enterprise integration patterns. 

Prometheus, Grafana, ELK/OpenSearch, Cloudera Manager monitoring, Airflow metrics, NiFi reporting tasks, or comparable enterprise tools. 

Linux, Java/JVM, Python runtimes, PostgreSQL or comparable metadata databases, networking, DNS, TLS, certificates, load balancing, nginx, firewalls, proxies, storage, identity management, and secrets management. 

SMAX, Jira, Confluence, Azure DevOps, Teams, or comparable tools used for incident, change, problem, release, and documentation management. 

• Business days (Monday–Friday), 8:30 AM–5:00 PM Central European Time. 

• Participation in planned maintenance windows, major incident support, release support, lifecycle activities, vulnerability remediation, security patching, disaster recovery tests, and on-call or standby coverage may be required for production and critical non-production services. 

We are committed to creating a diverse environment and are proud to be an equal opportunity employer. All qualified applicants receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status.