Site Reliability Engineer

Refer to the date listed at the top of this posting, if available. Continuous if date is blank.

Note: Applications will be accepted until 11:59 PM on the day prior to the posting end date above.

• Generous leave including 15 days of vacation, 15 days of sick leave and 11 paid holidays annually.
• A comprehensive Benefit Package with a generous benefit allowance to offset the cost of insurance premiums for employees and their eligible dependents.

• Full-time 40-hour work weeks
• Supports the Information Services Division
• Salary is up to $110,000 based on education and experience
• Position reports onsite in Oklahoma City
• Sponsorship: At this time, we cannot offer sponsorship now or in the future.

Incident Escalation & Remediation (approx. 40%)

• →Serve as the first technical escalation point for P1/P2 incidents affecting infrastructure, cloud, and application environments.
• →Lead hands-on troubleshooting across Windows and Linux servers, virtualization, Azure and hybrid cloud, networking (DNS, load balancing, firewalls), storage, databases, and application middleware.
• →Act as technical lead within the incident command structure, advising the Incident Commander on impact, options, and risk of remediation actions.
• →Engage and coordinate subject-matter experts from network, security, database, application, vendor, and agency teams; drive cross-domain remediation to closure.
• →Execute and document emergency changes in accordance with change management and audit requirements.

Observability Partnership (approx. 15%)

• →Collaborate with Observability engineers to define service-level indicators (SLIs), dashboards, and actionable alerts across Zabbix, Grafana, Azure Monitor, and Datadog.
• →Identify telemetry gaps surfaced during incidents and drive instrumentation improvements.
• →Reduce alert noise and false positives; tune thresholds to reflect true user and service impact.

Reliability Engineering & Problem Management (approx. 25%)

• →Lead blameless post-incident reviews; produce root cause analyses with corrective and preventive actions tracked to completion.
• →Define and maintain service-level objectives (SLOs) and error budgets with service owners and agency stakeholders.
• →Perform capacity planning, performance analysis, and resilience testing, including disaster recovery and failover validation.
• →Identify single points of failure and architectural risks; recommend and implement remediation.

Automation & Toil Reduction (approx. 15%)

• →Develop automated remediation, self-healing routines, and runbooks using PowerShell, Python, Bash, and infrastructure-as-code tooling (Terraform, Ansible, ARM/Bicep).
• →Measure and reduce operational toil; target a sustained ratio of engineering work to reactive work.
• →Contribute reliability guardrails to CI/CD pipelines in partnership with DevSecOps.

Governance, Security & Compliance (approx. 5%)

• →Ensure remediation activities comply with NIST SP 800-53 controls, CJIS Security Policy, IRS Publication 1075, HIPAA, and state security standards as applicable.
• →Preserve evidence and audit trails for incidents with security, privacy, or regulatory implications; coordinate with the security operations team and Operational Risk & Controls.
• →Support internal and external audits by providing incident, change, and availability documentation.

• This position works in a comfortable office setting with a computer and telephone for a large percentage of the workday.
• This position requires long periods, up to eight hours per day, of sitting and working at a desk. The noise level in the work environment is usually mild.
• Participation in a rotating on-call schedule, including occasional after-hours, weekend, and holiday response for critical incidents.
• Work supports essential public services; may be designated essential personnel during declared emergencies or continuity-of-operations events.
• Occasional travel to state data centers or agency sites.

•  Bachelor's degree in Computer Science, Information Systems, Engineering, or related field; equivalent professional experience may substitute year-for-year.
• 5+ years in infrastructure, systems, or application operations, including 2+ years in an SRE, production engineering, or senior escalation role.
• Demonstrated ability to troubleshoot complex, multi-layer incidents in enterprise or hybrid cloud environments under time pressure.
• Strong working knowledge of Windows Server, Linux, virtualization (VMware/Hyper-V), Microsoft Azure, and core networking (TCP/IP, DNS, DHCP, load balancing, firewalls).
• Hands-on experience with at least two observability platforms (e.g., Zabbix, Grafana, Azure Monitor, Datadog, Splunk).
• Scripting proficiency in PowerShell and/or Python; working knowledge of infrastructure as code.
• Working understanding of ITIL incident, problem, and change practices.
• Ability to pass fingerprint-based background checks and meet CJIS and IRS 1075 personnel security requirements.

• Experience in state, local, or federal government IT, or another regulated industry (insurance, financial services, healthcare).
• Experience defining and operating SLOs and error budgets for production services.
• Familiarity with containers and orchestration (Docker, Kubernetes/AKS) and with APM and distributed tracing.
• Experience with ServiceNow ITSM and incident command frameworks.
• Certifications such as Azure Administrator/Solutions Architect, AWS SysOps, CKA, ITIL 4, or Security+.
• Experience working within agile delivery (Kanban, Agile Release Train / PI planning).

The State of Oklahoma is an equal opportunity employer and does not discriminate on the basis of genetic information, race, religion, color, sex, age, national origin, or disability.

Current active State of Oklahoma employees must apply for open positions internally through the Workday Jobs Hub.

If you are needing any extra assistance or have any questions relating to a job you have applied for, please click the link below and find the agency for which you applied for additional information:

Agency Contact