Senior Cloud Engineer

At apexanalytix, we’re lifelong innovators! Since the date of our founding nearly four decades ago we’ve been consistently growing, profitable, and delivering the best procure-to-pay solutions to the world.   We’re the perfect balance of established company and start-up.  You will find a unique home here. 

Kubernetes (kubeadm, Cluster API, Kamaji, AKS) · Cilium + Hubble · Rook-Ceph and cephadm · FluxCD, Kustomize, Helm · KubeVirt · Harbor · Keycloak · OpenBao · OpenTofu and Ansible · Prometheus, Thanos, Grafana, Loki, Tempo · AlmaLinux, Ubuntu, Flatcar Container Linux · Gitea Actions · Go, Python, Bash

The Work

·        Run FluxCD as the only path to production — Kustomize layering, HelmReleases, and automated image updates — so that the fleet's desired state always lives in Git, not in someone's shell history.

·        Author and maintain Helm charts and distribute them through OCI registries.

·        Build infrastructure as code with OpenTofu/Terraform modules, Ansible roles, and cloud-init/Ignition templates.

·        Design and maintain CI/CD pipelines (Gitea Actions) and the Linux and Windows runner fleet behind them.

·        Automate the operational work that would otherwise be toil, in Go, Python, Bash, or PowerShell — favouring deterministic, re-runnable tooling over documented manual procedure.

·        Bring change discipline to a large fleet: a written blast-radius assessment before production changes, canary-then-fleet rollout, and a rollback path that has been thought through in advance.

·        Design, build, upgrade, and operate Kubernetes across bare-metal (kubeadm), virtualised (KubeVirt with Cluster API), and managed cloud (AKS) footprints.

·        Own the control plane: etcd backup, restore and defragmentation, API server tuning and audit logging, certificate rotation, and version upgrades under a clear support policy.

·        Operate Cluster API providers, machine deployments, MachineHealthChecks, and Kamaji hosted control planes.

·        Get workload topology right: PodDisruptionBudgets, anti-affinity and topology spread, requests and limits, autoscaling, and priority classes.

·        Build multi-tenancy that holds: namespaces, RBAC, admission webhooks, and Pod Security standards.

·        Diagnose the genuinely hard failures — kubelet and PLEG stalls, containerd faults, OOMKills, eviction storms, scheduler pressure — and turn each one into a runbook.

·        Operate Cilium as the fleet CNI: eBPF datapath, kube-proxy replacement, and endpoint and identity debugging.

·        Run the Cilium BGP Control Plane and LB-IPAM, advertising service IPs to the physical fabric and across clusters.

·        Build and troubleshoot ClusterMesh multi-cluster service discovery and its datapath.

·        Author CiliumNetworkPolicy at L3/L4/L7 and use Hubble to see east-west traffic as it actually flows.

·        Take on the advanced work: pod CIDR planning and renumbering, dual-stack IPv6 conversion, VRF-based pod isolation, and Multus secondary networks.

·        Own service exposure end to end — Gateway API and HTTPRoute, ingress, kube-vip, and CoreDNS tuning.

·        Operate Rook-Ceph and external cephadm clusters: OSD lifecycle, CRUSH and placement-group management, CephFS and RBD, rebalancing, and capacity planning.

·        Run the full CSI driver set — ceph-csi, NFS, SMB, NVMe-TCP, KubeVirt CSI, Portworx, and Pure Storage — and know which one belongs on which workload.

·        Manage StorageClasses, volume expansion, snapshots and clones, and RWO versus RWX topology decisions.

·        Own backup and disaster recovery for stateful workloads with Velero, Kopia, and etcd snapshots — and prove it by restoring, not by reading a green job status.

·        Investigate storage performance properly: PVC benchmarking, I/O latency analysis, and noisy-neighbour isolation on shared arrays.

·        Own Harbor as the enterprise registry: projects and quotas, retention, cross-site replication, garbage collection, Trivy scanning, robot accounts, and high availability.

·        Own Keycloak as the identity provider: realms, clients, scopes and mappers, OIDC for kubectl, Grafana and Headlamp, identity brokering and IdP federation, group-to-role mapping, and Infinispan multi-site caching.

·        Run the secrets platform: OpenBao raft clusters, External Secrets Operator, rotation, and SSH certificate authority issuance.

·        Automate certificates and supply-chain trust: cert-manager and Let's Encrypt, wildcard rotation, registry mirroring, and image signing and policy enforcement.

·        Support the data and middleware tier the platform depends on: PostgreSQL, OceanBase, Valkey/Redis Sentinel, and RabbitMQ.

·        Operate Prometheus, Thanos, Grafana, Loki, Tempo, and Alloy, and author the dashboards and alert rules the team actually relies on.

·        Take part in the on-call rotation, act as incident commander during platform outages, and write the root cause analysis afterwards.

·        Design and test disaster recovery and failover — including the maintenance window, the rollback plan, and the report that follows.

·        Own capacity planning and hardware and software end-of-life tracking, and lead OEM and vendor escalations.

·        Work within a compliance-regulated environment (FedRAMP, SOC 2): segmentation, audit logging, and change evidence.

·        Mentor engineers earlier in their careers, lead design reviews, and raise the team's operating standard.

·        Automate the path from a new server to a production Kubernetes node so provisioning is repeatable rather than manual — PXE, kickstart, cloud-init, and Ignition.

·        Provision and harden node operating systems: AlmaLinux 9, Ubuntu 24.04, Flatcar Container Linux, and Windows Server, including kernel and sysctl tuning, NUMA alignment, and container runtime configuration.

·        Configure host networking and storage for new capacity: LACP bonding, VLAN trunks, nmstate policies, MTU consistency, LVM, multipath, and NVMe-TCP.

·        Bring nodes into the cluster with kubeadm or Cluster API, then validate CNI, CSI, and workload readiness before promoting them to production.

·        Partner with datacenter operations and remote hands on new hardware — commissioning, out-of-band management (iDRAC, iLO, IPMI, UCSM), and firmware and BIOS lifecycle — and confirm fabric readiness before rack-in.

·        Retire and rotate nodes safely: HA-floor analysis, controlled drain, reimage, and return to the pool with no workload impact.

The Must-Haves

·        More than 8 years in infrastructure engineering, including at least 5 years running production Kubernetes at scale.

·        Deep Kubernetes knowledge: control-plane internals, etcd operations, cluster upgrades, RBAC and admission control, scheduling, and real failure diagnosis rather than restart-and-hope.

·        A strong GitOps practice — FluxCD or ArgoCD with Kustomize and Helm — and the conviction that Git should be the only path to production.

·        Substantial CNI experience with Cilium: eBPF datapath, BGP control plane, LB-IPAM, ClusterMesh, network policy, and Hubble. Experience with other CNIs is welcome, but Cilium is our standard.

·        Substantial CSI and storage experience, ideally with Ceph (Rook or cephadm): OSD and CRUSH management, CephFS and RBD, snapshots, and performance troubleshooting.

·        Production ownership of a container registry (Harbor or equivalent) and an identity provider (Keycloak or equivalent) — or the depth and appetite to take both on.

·        Strong Linux fundamentals across RHEL/AlmaLinux, Ubuntu, and Flatcar: systemd, kernel and sysctl tuning, the storage and network stacks, and performance debugging.

·        Comfort working close to the hardware — bare-metal provisioning, out-of-band management, and firmware lifecycle — and the instinct to automate it rather than repeat it.

·        Automation skill in OpenTofu/Terraform, Ansible, and Bash, plus at least one of Python or Go.

·        Hands-on experience with Cluster API and KubeVirt for automated cluster and virtual-machine provisioning.

·        Working knowledge of datacenter networking — VLANs, LACP, BGP, VXLAN, MTU and jumbo frames — enough to work fluently alongside the network team.

·        Experience with backup and disaster recovery tooling (Velero, Kopia, etcd snapshots), including having actually executed restores.

·        Experience running observability at scale: Prometheus, Grafana, Thanos, Loki, and sensible alert tuning.

·        Certified Kubernetes Administrator (CKA), or equivalent demonstrable depth; CKS or CKAD is a plus.

·        A bachelor's degree in Computer Science, Information Technology, or equivalent practical experience.

·        Clear writing: design documents, runbooks, and root cause analyses that a colleague can act on without asking you first.

·        An ownership mindset, accountability for uptime, and the generosity to bring other engineers up with you.

·        Experience building and operating bare-metal Kubernetes, not managed cloud only.

·        Azure experience (AKS and hybrid connectivity); AWS or GCP also useful.

·        Experience supporting Windows Server worker nodes in Kubernetes.

·        Experience in a FedRAMP, SOC 2, or comparable regulated environment.

·        Red Hat (RHCSA/RHCE), Ceph, Isovalent/Cilium, HashiCorp Vault, or Azure (AZ-104 / AZ-305) certification.

·        Contributions to open-source infrastructure projects.