Senior Software Engineer, Ingestion Team

We’re a team of AI, technology, and language experts whose DNA lives in Alexa, Siri, Watson, and virtually every human language technology product on the market. Now we’re building an industry-leading knowledge management and Retrieval-Augmented Generation (RAG) platform. Our proprietary, cutting-edge natural language processing capabilities transform unstructured data into meaningful experiences that increase productivity with unmatched accuracy and speed.

The Ingestion team is responsible for everything that happens between content arriving from a connector and that content being ready for search and retrieval. This means document processing pipelines that handle parsing, text extraction, chunking, metadata enrichment, embedding generation, and index population — across every file format and content type our customers throw at us.

We’re in the middle of a significant architectural evolution — migrating from a legacy pipeline to a modern, workflow-orchestrated architecture with cleanly separated processing stages: intake, transformation, enrichment, and indexing. The team is also actively designing the next iteration of the pipeline to push further on throughput and resilience.

This is real systems engineering: the problems are about scale, reliability, and the messy realities of processing millions of documents with wildly different structures.

Design and build pipeline stages for our modern ingestion architecture - from document intake through embedding generation and index writing

Contribute to the design of next-generation pipeline architecture as the system evolves

Improve system stability and scale: identify bottlenecks, reduce failure rates, and build observability into every stage

Work with workflow orchestration tools to manage complex, multi-step document processing with retry logic, error handling, and state management

Handle the realities of document diversity: PDFs, HTML, Office formats, images, structured and semi-structured data - all flowing through the same pipeline

Collaborate with the Connectors team (upstream) and Retrieval team (downstream) to ensure data flows cleanly across system boundaries

Participate in the ongoing migration from legacy systems, balancing new development with operational stability