26 Jul 2024 AI Engineering5 min read

ColPali: Visual Document Retrieval Without OCR

ColPali (June 2024) replaces the OCR and parsing pipeline by embedding document page images directly. We explain the late-interaction mechanism inherited from ColBERT, put the ViDoRe results — 81.3 versus 66.1 nDCG@5 — into context, and assess the storage and scaling trade-offs and the role of visual retrieval in production RAG systems as of July 2024.

Where text-only RAG silently fails

A retrieval-augmented generation system succeeds or fails with its retriever, yet most production systems search only extracted text. A PDF is parsed, processed with OCR, split into chunks, and embedded. Anything the parser cannot translate into text disappears before the first query arrives. Tables lose alignment, stamps and signatures vanish, and technical drawings become empty strings. The failure remains invisible: the system keeps returning plausible passages while the decisive page never entered the index.

This problem is most acute in the corpora where retrieval quality matters most. Invoices, scanned contracts, engineering drawings, regulatory filings and slide decks carry a large share of their information in layout, figures and visual marks. In our project work at Blue IT Systems we regularly see document sets where the pages a domain expert would pick are exactly the pages a text pipeline cannot represent. Recall failures of this kind rarely show up in evaluation sets built from clean text.

Documents are chunked, embedded and indexed — vectors plus keywords. 1/4

What a parsing pipeline actually costs

A modern ingestion pipeline is a chain of models: layout detection, OCR, table reconstruction, chunking, and optionally captioning of figures with a vision-language model. Each stage has its own error modes, and errors compound. The ColPali authors measure a representative pipeline built on the Unstructured tool — layout detection, OCR, captioning, embedding — at about 7.22 seconds per page on an NVIDIA L4 GPU.

The deeper problem is not speed but information loss. In the common text-only configuration, figures, images and tables are treated as noise and filtered out. The pipeline does not fail loudly on a construction drawing or an approval stamp. It simply indexes nothing.

ColPali embeds the page as an image

ColPali (Faysse et al., arXiv:2407.01449, first version 27 June 2024) removes the pipeline. Each document page is rendered as an image and fed to a vision-language model — PaliGemma-3B, released by Google in May 2024. The SigLIP vision encoder splits the page into 1,024 patches, the patch embeddings pass through the Gemma-2B language model, and a projection layer maps every output token to 128 dimensions.

The result is a multi-vector representation: roughly one thousand contextualized 128-dimensional vectors per page. No OCR, no layout detection, no chunking, no captioning. Indexing takes 0.39 seconds per page on the same L4 GPU — about eighteen times faster than the parsing pipeline, in a single forward pass with fixed sequence length.

The enabling observation is subtle: after multimodal fine-tuning, a VLM's image-token embeddings live in the same latent space as its text-token embeddings. A textual query can therefore be matched directly against visual patches. The authors' contrastive fine-tuning makes that alignment sharp enough for retrieval.

Late interaction does the matching

Scoring follows ColBERT (Khattab and Zaharia, 2020). A query is embedded into one vector per token. For each query token, the late-interaction operator takes the maximum dot product over all patch vectors of a page and sums these maxima into the page score. The rich token-to-patch interaction happens at query time; the expensive page encoding happens offline.

Despite the richer representation, runtime cost remains modest. Query encoding with ColPali's language model takes about 30 ms, versus 22 ms for a BGE-M3 text encoder; late-interaction scoring adds roughly 1 ms per 1,000 pages in the corpus. Because the operator is fully differentiable, the entire retriever can be trained end to end. The released model was fine-tuned with LoRA adapters on about 100,000 query-page pairs.

ViDoRe puts numbers on the gap

Because existing benchmarks evaluate text embeddings on clean text, the authors built ViDoRe, the Visual Document Retrieval Benchmark: ten page-level retrieval tasks in English and French, spanning academic datasets (DocVQA, InfoVQA, TAT-DQA, arXivQA, TabFQuAD) and practical domains such as energy, government, healthcare and AI. The main metric is nDCG@5. The benchmark scores three practical requirements together — retrieval quality, query latency and indexing throughput — a framing we consider more honest than embedding-only leaderboards.

The gap is widest where text pipelines discard the most: infographics (InfographicVQA), figures (arXivQA) and tables (TabFQuAD). Notably, ColPali also retrieves text-centric documents better than the text baselines across all evaluated domains and languages — the advantage does not rest on visual edge cases alone.

System	ViDoRe average nDCG@5	Indexing per page (NVIDIA L4)
Unstructured + OCR + BM25	65.5	≈7.22 s (full pipeline)
Unstructured + OCR + BGE-M3	66.1	≈7.22 s (full pipeline)
ColPali (late interaction)	81.3	0.39 s

What ColPali does not solve

The multi-vector representation has a price. Storing 1,024 vectors of 128 dimensions at 16-bit precision costs roughly 260 KB per page; a million-page corpus needs on the order of 260 GB, against a few kilobytes per chunk for single-vector embeddings. Naive late-interaction scoring scans every page; scaling to millions of documents requires optimized engines such as PLAID, and most vector databases offer no native multi-vector support today.

ColPali is a retriever, not a reader. It returns page images; answering still requires a vision-language model downstream, and page-level granularity means no sub-page citations out of the box. The 3B-parameter encoder needs a GPU at indexing and query time. Training data is dominated by English, with French covered; other languages are unverified. And the paper is four weeks old — independent replication is still pending.

Outlook from July 2024

We expect the pattern to outlive the checkpoint. The backbone is replaceable, and better small VLMs will lift retrieval quality without changing the architecture. The obvious engineering targets are storage — pooling and compressing patch vectors — and native multi-vector support in vector databases. The public ViDoRe leaderboard will make progress measurable.

The authors make a broader bet: end-to-end RAG in vision space, where a VLM answers directly from retrieved page images and extracted text never exists. We consider that plausible for archives where parsing already fails — drawings, stamps, and scans. OCR pipelines will not disappear; whenever extracted text is itself the deliverable, they remain necessary. For retrieval, however, the page image has become a serious alternative. We are evaluating it where conventional pipelines demonstrably lose recall.