Medior AI Engineer

Keystone Solutions is seeking a Medior AI Engineer to join our consultancy mission at a client site. You will be part of the AI Center of Excellence, focusing on the Run phase of AI services in production. This role involves ensuring the reliability, security, and cost-efficiency of AI services built on a hybrid operating model, utilizing both managed Azure services and self-hosted open-source components.

• →Operate the AI services in production: monitor availability, latency, and errors, handle incidents and service requests, analyze root causes, then fix or escalate.
• →Keep answer quality under control: run and maintain evaluation sets with business content owners, track answer quality, non-answer rate, and user feedback, and detect and investigate quality regressions.
• →Operate the ingestion pipelines: manage incremental refresh, failed documents, index health, and onboarding of new content sources or business domains.
• →Operate agent integrations: monitor tool and integration calls to business systems, handle failures and timeouts, and review action logs and exceptions.
• →Manage change safely: model version upgrades and retirements, prompt and configuration changes, and dependency and security updates, validated against evaluation sets before release.
• →Follow run cost: monitor token consumption and cost per service, alert on drift, and propose optimizations.
• →Maintain Terraform, Helm, and Azure DevOps artifacts, ensuring consistency across DEV, TST, ACC, and PROD environments.
• →Apply data protection rules: manage trace retention, erasure, and access restrictions as defined in each service’s data protection impact assessment.
• →Maintain runbooks and documentation, and report service health to service owners and the AI CoE regularly.
• →Deliver small enhancements and fixes; structural changes and new features will be handled by the Build team.

• Azure DevOps - Level: Intermediate - Most recent: This year
• Azure Kubernetes Service - AKS - Level: Intermediate - Most recent: This year
• Azure OpenAI Service - Level: Intermediate - Most recent: This year
• LangChain - Level: Intermediate - Most recent: Any time
• LLM / Large Language Models - Level: Intermediate - Most recent: This year
• Python - Level: Intermediate - Most recent: This year
• RAG (Retrieval-Augmented Generation) - Level: Intermediate - Most recent: This year
• Terraform - Level: Intermediate - Most recent: This year
• Vector databases - Level: Intermediate - Most recent: This year

Dutch or French
Level Native or bilingual proficiency
English
Level Full professional proficiency