LLM Platform Engineer

Read more at https://allspice.io

This is a high-impact, cross-functional role at the intersection of engineering, data, and customer experience. You will build and maintain the internal tooling, analytics platforms, and customer-facing applications that power AllSpice's Generative AI team.

You will:

• Collaborate closely with product and engineering teams to understand workflows and build tools that solve real problems.

• Own the evaluation and observability layer for our AI products — building dashboards, analytics pipelines, and reporting tools that give the Gen AI team and stakeholders visibility into model performance, usage, and customer outcomes.

• Develop customer-facing tooling and lightweight web applications that surface AI-powered insights and enable seamless interaction with AllSpice's products.

• Build and maintain scripts, data pipelines, and integrations that connect internal systems, third-party services, LLMs, and MCPs.

• Write clear documentation and communicate findings to both technical and non-technical audiences, including leadership and customers.

• Champion code quality, testing, and reliability across the tooling stack through reviews, mentorship, and thoughtful design.

• Build interactive internal dashboards to track DRCY performance metrics, customer usage patterns, and model quality over time.

• Develop a lightweight web application that allows customers to explore AI-generated design review results, drill into findings, and provide feedback.

• Create and maintain data pipelines that aggregate evaluation results, customer feedback, and telemetry into a unified analytics layer.

• Build scripting tools and automations to streamline the Gen AI team's development and testing workflows (e.g., batch evaluation runs, report generation).

• Design and implement RAG-based information repositories and tooling to capture component data and domain knowledge for use by our suite of AI products.

• Integrate with CI/CD systems (GitHub Actions) to surface AI insights directly in engineering workflows.

• Work with infrastructure to maintain appropriate reporting for centralized cache service.

• Languages: Python, TypeScript/JavaScript, SQL

• Analytics & Data: SQL databases (Postgres), Jupyter Notebooks, business intelligence/analytics tools

• AI/LLM: Langchain, Pydantic AI, Langfuse

• Web: Vue/TypeScript front-end, common web architectures and APIs

• Infrastructure: GitHub Actions for CI/CD, Docker Swarm & Terraform, AWS

• Our Hub application is a soft fork of Gitea

  • Go [server-side]

  • Vue/TypeScript Front-end

Our ideal candidate should have the following:

• 4+ years of software engineering experience with a focus on tooling, analytics, or full-stack web development.

• Strong SQL skills and experience working with data/analytics tools (e.g., Jupyter, Metabase, Looker, or similar BI platforms).

• Proficiency in Python scripting and automation for data pipelines, integrations, and internal tools.

• Experience building web applications or customer-facing tools using modern frameworks (e.g., Vue, React, TypeScript).

• Familiarity with LLM-enabled applications and frameworks such as Langchain, including prompt construction and evaluation best practices.

• Excellent communication skills — you can translate technical findings into clear insights for engineers, product managers, and customers alike.

• A customer-first mindset: you think about the end-user experience and proactively seek feedback to improve the tools you build.

• Bachelor's degree or higher in a technology-related field, or equivalent practical experience.

• Project management skills with the ability to lead and collaborate with varied stakeholders to move projects forward.

• Comfort with ambiguity, autonomy, and being self-directed in a fast-moving environment.

• (Preferred) Availability to work out of our flex offices in San Francisco or Boston on a hybrid basis (2–3 times/week), but we are flexible for the right candidate.

You don't necessarily have to be proficient in all of these, but the more you know, the better:

• Python, TypeScript, SQL

• Business analytics and data visualization tools

• Langchain, Pydantic AI, and LLM evaluation frameworks

• GitHub Actions development and CI/CD

• Common web architectures, APIs, and front-end frameworks (Vue, React)

• Monitoring and observability tools (e.g., Langfuse, DataDog, Grafana)

• Project management and stakeholder communication