Senior AI Software Engineer - Model Evaluation (f/m/d)

At Aleph Alpha, we foster a culture built on ownership, autonomy, and empowerment. Teams and individual contributors are trusted to take responsibility for their work and drive meaningful impact. We maintain a flat organizational structure with efficient, supportive management that enables quick decision‑making, open communication, and a strong sense of shared purpose.

As a Senior AI Engineer in Pre-training Evaluation, you will work across the full stack of evaluation - from methodology design to implementation to analysis. Some weeks you'll be deep in benchmark curation, understanding what a given eval actually measures and whether it predicts downstream performance. Other weeks you'll be optimising pipeline throughput or building dashboards that surface training signals.

We are looking for someone that combines significant research experience (in industry or academia) with high engineering competence.

Your work sits at high leverage: the evaluations you design and build determine which training runs we pursue, which data mixtures we prioritise, and how we allocate compute. You'll have direct influence on the models we ship.

• →

• Experience with LLM evaluation, benchmark design, evaluation dataset curation, and experimental design.

• Familiarity with statistical methods for evaluation and experiment design.

• Track record of shipping impactful technical work - whether that's research, infrastructure, or both.

• Strong Python skills and comfort with ML tooling (PyTorch, evaluation frameworks, distributed systems).

• Ability to reason about what an evaluation measures and whether it matters - not just run benchmarks, but understand them.

• Ownership mentality: you see problems through from diagnosis to solution to deployment.

• Willingness to relocate to Heidelberg or travel regularly (potentially weekly).

• Understanding of foundation model training - how data, scale, and architecture affect capabilities.

• Experience with large-scale data processing or ML infrastructure.

• German language proficiency (helpful for evaluating German capabilities, not required).

• PhD in machine learning, NLP, statistics, or a related field (valued but not required - we care about what you can do).