Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

This technical report introduces the Gemini 1.5 family, a generation of highly compute-efficient multimodal models designed to recall and reason over fine-grained information from millions of tokens of context, spanning multiple long documents and hours of video and audio. The family comprises an updated Gemini 1.5 Pro, which exceeds the prior version on most capabilities and benchmarks, and Gemini 1.5 Flash, a lighter variant built for efficiency with minimal quality regression.

Gemini 1.5 models achieve near-perfect recall on long-context retrieval across modalities, improve the state of the art on long-document QA, long-video QA, and long-context ASR, and match or surpass Gemini 1.0 Ultra across a broad benchmark suite. Studying the limits of its long-context ability, the authors find continued gains in next-token prediction and greater than 99% retrieval up to at least 10M tokens, far beyond contemporaries. Real-world uses show substantial time savings, and remarkably, given only a grammar manual the model learns to translate English to Kalamang, a language with fewer than 200 speakers, at a human-comparable level.