Principal Linux Engineer

POSITION SUMMARY:

The Principal Linux Engineer is the senior technical authority for the enterprise Linux platform and serves in a Linux support lead capacity, owning the design, build standards, hardening, and day-to-day reliability of the Linux server estate. The role additionally supports the Debian-based Linux operating system (Debian 13 "trixie" on ARM hardware) that ships embedded in certain company products, partnering with Product Engineering and R&D on OS image build, hardening, patching, and field escalations. This role sets engineering, scripting, and automation standards across both estates and acts as the final escalation point for complex Linux incidents and root-cause analysis.

This position manages the Linux support queue and on-call rotation, mentors junior and mid-level engineers, and partners with Security, Network, Database, Applications, and Quality teams to sustain systems available, secure, compliant, and audit-ready in a 24x7x365 environment.

DUTIES AND RESPONSIBILITIES: 

 Serve as technical lead and final escalation point for Linux support, owning triage, root-cause analysis, and permanent corrective action for complex incidents.

 Own the architecture, build standards, and lifecycle of the Linux estate (RHEL and derivatives), including capacity planning and hardware/OS refresh.

 Define and enforce configuration management and infrastructure-as-code practices using Ansible, Terraform, Git, and CI/CD pipelines.

 Provide engineering and support ownership for the Debian Linux OS distributed as part of company products, including image definition, kernel and device-tree configuration, package selection, hardening, and release qualification.

 Partner with Product Engineering, R&D, Manufacturing, and Field Service on Raspberry Pi image provisioning at build, unattended upgrade and patch delivery mechanisms, and escalated field issues.

 Lead performance analysis, fault isolation, and reporting of service exceptions across the Linux estate

 Partner with Network, Security, Windows, Database, and Application teams on data center builds, migrations, and cross-platform projects.

 Mentor and develop Linux engineers and administrators, providing technical direction, training, and workload prioritization.

 Author and maintain SOPs, runbooks, and change/configuration records in line with company quality and regulatory requirements, and perform other duties as assigned by the organization.

EXPERIENCE AND QUALIFICATIONS:

 Bachelor's degree in Computer Science, Information Systems, or equivalent work experience.

 A minimum of ten years of progressive Linux systems administration/engineering experience in production data center environments, that includes a minimum of three years in a lead, principal, or equivalent senior technical capacity.

 Experience building, hardening, and supporting an embedded or appliance Linux OS that ships as part of a product, including reproducible image creation (debootstrap, pi-gen, rpi-image-gen, or equivalent), first-boot provisioning, and read-only or A/B root filesystem strategies.

 Hands-on experience supporting databases on Linux: Oracle, MySQL/MariaDB, PostgreSQL, MongoDB.

 Experience working in a production data center environment, including clustering, high availability, and disaster recovery for Linux workloads.

 Experience operating in a regulated environment (GxP, ISO, or similar) with change control, documentation, and audit support.

 Experience with cloud hosting, SaaS, PaaS technologies – AWS experience preferred.

 Red Hat certification (RHCSA/RHCE) strongly preferred; cloud, security, or ITIL certifications a plus.

KNOWLEDGE, SKILLS AND ABILITIES:

 Expert-level knowledge of enterprise Linux administration and internals:

o Distributions: RHEL, CentOS/Rocky/AlmaLinux, Debian (including Debian 13 "trixie"), Ubuntu

o Kernel tuning, systemd, LVM, filesystems, networking, and performance diagnostics

 Security hardening and tuning of Linux servers: CIS benchmarks, SELinux, and vulnerability remediation.

 Expert knowledge of installation and configuration procedures of physical and virtual servers.

 Expert knowledge of patch and errata management at scale (Red Hat Satellite, Foreman, or equivalent), and backup and restore operations.

 Expert knowledge of configuration management and automation at scale: Ansible required; Terraform for infrastructure-as-code provisioning; Git-based version control and CI/CD pipelines.

 Hands-on Debian expertise, specifically Debian 13 ("trixie") on Raspberry Pi: APT and dpkg package management, .deb packaging, apt pinning and preferences, unattended-upgrades, and local/private APT repository management.

 Working knowledge of ARM64/aarch64 Linux and Raspberry Pi platform specifics: boot chain and firmware, device tree overlays, GPIO/I2C/SPI and serial peripherals, SD/eMMC endurance, and thermal and power constraints.

 Familiarity with product OS release qualification, remote upgrade and patch delivery to fielded devices, rollback strategy, and CVE tracking against a shipped software bill of materials.

 VMware vSphere and container platforms (Docker, Kubernetes/OpenShift) experience preferred.

 Expert in Shell and Python scripting for automation, tooling, and diagnostics; Perl a plus.

 Strong understanding of DNS, HTTP/S, LDAP, SSH, NFS, Kerberos, SMTP, and SNMP, including Active Directory integration of Linux hosts.

 Understanding of basic Network concepts and the OSI model.

 Familiarity with web related technologies (Web applications, Web Services, Service Oriented Architectures) and of network/web related protocols.

 Ability to provide day to day guidance/mentoring to others in a consistently supportive and collaborative approach.

 Advanced problem-solving and root-cause analysis skills, with composure while leading major incident response.

 Strong verbal and written communication skills, including the ability to brief senior leadership on risk, incidents, and roadmap.

 Must be able to move/lift up to 50 pounds.

PHYSICAL DEMANDS:

This position requires the ability to work standing up in data centers, data closets and other secure environments, along with the ability to lift moderately heavy equipment when required.

WORK ENVIRONMENT:

This position is performed in a traditional office environment. Some local travel between office sites.

MSD is an Equal Opportunity/Affirmative Action Employer. We are committed to fostering a diverse and inclusive workplace where all individuals are treated with respect and dignity. We welcome applications from all qualified candidates, making employment decisions without regard to race, color, religion, creed, sex, sexual orientation, gender identity, genetic information, marital status, national origin, age, protected veteran status, pregnancy, disability status, or any other protected characteristic. For our full EEO/AA and Pay Transparency statement, please visit here. Meso Scale Diagnostics uses E-Verify to validate the work eligibility of candidates.