Machine Learning Engineer, Speech - Joint Audio-Video Modeling

Cantina Labs is a social AI company, developing a suite of advanced real-time models that push the boundaries of expression, personality, and realism. We bring characters to life, transforming how people tell stories, connect, and create. We build and power ecosystems. Cantina, our flagship social AI platform, is just the beginning.

If you're excited about the potential AI has to shape human creativity and social interactions, join us in building the future!

We're looking for a Research / ML Engineer to join our Speech Team to build state-of-the-art speech and audio generation systems end-to-end from data specs through production inference with a focus on joint audio-video modeling.

You'll own the audio side of multimodal generation: the representations (audio VAEs, neural codecs), the generative backbone (diffusion / flow-matching transformers), and the conditioning and alignment machinery that makes characters speak, sing, and emote in sync with what's on screen. That includes voice cloning and multi-speaker conditioning inside joint AV models, cinematic dialogue with music and sound design, and adjacent speech tasks (controllable TTS, voice conversion) that feed the same stack.

You'll drive the model ↔ data ↔ eval flywheel, partnering closely with research, video, data, and infra to ship fast, reliable, and cost-aware models. In this role you'll work at the intersection of cutting-edge research and practical engineering, contributing to the development of safe, steerable, and trustworthy AI systems.

You will thrive in this role if you:

• See research and engineering as two sides of the same coin and enjoy owning work end-to-end.

• Are excited to work across modalities and collaborate closely with a video generation team rather than staying inside audio.

• Are results-oriented, flexible, and willing to pick up whatever moves the needle.

• Like collaborating closely with infra, data, and product to ship measurable improvements.

• Enjoy designing experiments, listening tests, and metrics that correlate with user-perceived quality.

• Are eager to learn every day, and to find and solve unique large-scale problems.

• →
  • →

    Exceptional research/development experience with large-scale audio models (>8B parameters, >500k hours of data).

  • →

    Deep hands-on experience with diffusion and/or flow-matching transformers, including practical knowledge of samplers, schedules, conditioning mechanisms, and distillation.

  • →

    Deep hands-on experience training audio VAEs, neural audio codecs, and vocoders latent/tokenizer design, reconstruction and perceptual objectives, adversarial training.

  • →

    Strong experience with multi-node, multi-GPU distributed training (FSDP/DeepSpeed or equivalent).

  • →

    Strong software engineering skills with a proven track record of building complex systems.

  • →

    Strong with PyTorch and performance work (profiling, CUDA/Triton/C++ as needed) and writing reliable production-quality code.

  • →

    Shipped large-scale speech/audio or multimodal generative models to production.

  • →

    Background in working with large-scale ML data, and the ability to iterate on data and triangulate quality using both subjective and objective signals.

  • →

    Experience with voice cloning, speech control/steerability, or expressive speech generation.

  • →

    Notable publications and/or open-source contributions in speech/audio/ML.

  • →

    Strongly preferred:

    • →

      Experience with multimodal audio-video modeling: joint AV generation of multi-shot, multi-speaker scenes with dialogue, music, and sound design generated jointly with video, and the cross-modal alignment that keeps them in sync.

    • →

      Experience with video generation: video diffusion/flow-matching transformers, video VAEs, conditioned and multi-shot generation, building data pipelines for video models.

    • →

      Streaming or real-time generation, causal distillation (e.g., Self Forcing / Self Forcing++).