AI Research Engineer

As an AI Research Engineer in the TrainAI research team, you will bridge applied research and engineering execution. Your core focus will be designing, building, and scaling continuous LLM evaluation pipelines, experiment orchestration tools, and automated pre-annotation workflows. Working directly with the Head of Innovation and data engineering teams, you will turn cutting-edge multilingual evaluation methodologies into reliable, automated research and production tools.

• →Evaluation Pipelines & Benchmarking: Design, build, and maintain automated execution pipelines for continuous LLM benchmarking (such as TrainAI Bench) across diverse languages, tasks, and models.
• →Pre-Annotation & QA Automation: Develop and roll out LLM-assisted pre-annotation and quality assurance pipelines to automate production data annotation, lowering costs and improving data consistency.
• →LLM-as-a-Judge Frameworks: Implement and refine automated evaluation methodologies, including LLM-as-a-judge pipelines, custom evaluation metrics, and scoring algorithms.
• →Experiment Orchestration & Research Support: Build experiment infrastructure, datasets, and execution pipelines to support flagship multilingual LLM research studies and thought leadership reports.
• →Research-to-Engineering Bridge: Translate applied NLP research into scalable prototypes and work alongside the core data engineering team to integrate research tools into production operations.
• →Thought Leadership & Publication: Collaborate with the research team on writing up research findings, benchmark reports, and thought leadership pieces for internal and public release.
• →Emerging AI Methods Exploration: Prototype novel capabilities in emerging areas such as agentic AI evaluation, synthetic data quality certification, and domain-specific LLM assessment.
• →Documentation & Best Practices: Maintain comprehensive documentation, code bases, and technical guides for TrainAI evaluation frameworks and tools.

• Education & Experience: Master’s degree (MSc or equivalent) in Computer Science, Artificial Intelligence, Computational Linguistics, or a related technical field, combined with 3–5+ years of hands-on experience in NLP or ML engineering.
• Programming & ML Stack: Advanced proficiency in Python and deep learning/NLP frameworks (PyTorch or TensorFlow, Hugging Face transformers).
• LLM Tooling & Frameworks: Proven experience with LLM orchestration frameworks (e.g., LangChain, LlamaIndex) and automated evaluation techniques (e.g., LLM-as-a-judge, benchmarking harnesses).
• Data & Analytics Libraries: Expertise in data manipulation, structured evaluation data processing, and analysis using pandas, NumPy, or related tools.
• Cloud Infrastructure: Practical experience deploying and managing ML workflows on cloud platforms (Azure, AWS, or Vertex AI).
• System Design: Experience building scalable data or experiment orchestration pipelines.
• Familiarity with multilingual NLP, machine translation, or computational linguistics (e.g., cross-linguistic variations, syntax, morphology).
• Knowledge of localization workflows, annotation platforms, or human-in-the-loop data pipelines.
• Track record of contributing to open-source NLP projects, evaluation frameworks, or published technical benchmarks.

Get the 3Ps right – Partner, Pioneer, Progress – and we´ll Deliver together as RWS.

Recruitment Agencies: RWS Holdings PLC does not accept agency resumes.  Please do not forward any unsolicited resumes to any RWS employees.  Any unsolicited resume received will be treated as the property of RWS and Terms & Conditions associated with the use of such resume will be considered null and void.