The Best Embedding Model for RAG Is the One That Wins on Your Data

Teams often choose an embedding model by opening a public leaderboard, selecting one of the top names, and starting to index their documents.

Six months later, the problems appear.

Exact product codes are missing from search results. Multilingual queries retrieve the wrong policy. Similar but legally different clauses are placed next to each other. Infrastructure costs rise as the index grows. Then the team discovers that changing the model means embedding the entire corpus again and rebuilding the vector index.

The original model may not have been bad. The decision process was incomplete.

The right embedding model is not the model with the highest general benchmark score. It is the model that retrieves the correct evidence from your data, within your latency, cost, security, and operational constraints.

What an embedding model actually controls

An embedding model converts text, code, images, or other content into numerical vectors. A retrieval system compares these vectors to identify content that is semantically related to a user's query.

In a typical RAG pipeline, the embedding model sits inside a larger sequence:

Document ingestion - parsing - chunking - embedding - indexing - retrieval - reranking - generation

A weak embedding model can prevent the right evidence from reaching the language model. But a strong model cannot compensate for every other architectural problem.

For example: poor parsing can remove headings and table relationships; oversized chunks can combine unrelated facts; undersized chunks can destroy necessary context; missing metadata can make access filtering impossible; weak reranking can place superficially similar content above the correct answer; and an LLM can still misinterpret properly retrieved evidence.

This is why embedding selection should be treated as an architecture decision, not a model-shopping exercise.

Why public leaderboards are useful-and insufficient

MTEB was created because embedding models were previously evaluated on narrow, incompatible sets of tasks. It covers retrieval, clustering, classification, reranking, semantic similarity and other workloads. The benchmark's original findings showed that no single method dominated every task. MMTEB later expanded multilingual and domain coverage significantly. (arXiv)

A leaderboard can answer:

Which models deserve to enter our test?

It cannot reliably answer:

Which model should run our production system?

A public dataset rarely reproduces your combination of: internal terminology; abbreviations; product identifiers; document structure; languages; duplicate content; incomplete user questions; access restrictions; and industry-specific distinctions.

For RAG, the relevant benchmark category is usually retrieval. Even then, an average retrieval score can conceal serious weaknesses in the exact query types that matter to your business.

What should drive the model decision

Retrieval quality on your corpus

The model must find the evidence required to answer the actual question - not merely a document with similar vocabulary.

Your evaluation set should include several query classes:

A model that performs well on common questions but fails on negative conditions or identifiers may be unsuitable for legal, financial or operational search.

Domain and language fit

"Multilingual" is not a binary property.

A model may support a language technically while performing poorly on: specialist vocabulary; inflected forms; transliterated names; mixed-language documents; industry abbreviations; and queries written in one language against documents in another.

For multilingual systems, include native-language and cross-language queries in the gold set. Do not infer performance from English MTEB results.

Qwen3 Embedding supports more than 100 languages and configurable dimensions up to 4096. BGE-M3 supports dense, sparse and multi-vector retrieval and processes inputs up to 8192 tokens. Snowflake Arctic Embed 2.0 was designed for multilingual retrieval under an Apache 2.0 licence. These are useful shortlist candidates, not automatic winners. (Hugging Face)

Query and document behavior

Some models support separate query and document modes or task-specific instructions.

This matters because a search query and a source passage serve different functions: the query expresses an information need and the document contains potential evidence.

Using the wrong input mode can reduce retrieval quality even when the model itself is strong.

The evaluation must therefore replicate the exact production configuration: correct query prefix or instruction; correct document mode; same normalization; same output dimension; and same similarity metric.

Testing a model with default settings and deploying it with another configuration invalidates the comparison.

Vector dimensions and index cost

Larger vectors are not automatically better.

They can increase: raw storage; index memory; network transfer; distance-computation cost; backup size; and re-indexing time.

Several managed models now support flexible dimensions. OpenAI's third-generation embedding models default to 1536 and 3072 dimensions, while the API can shorten their output. Cohere Embed v4 supports 256-1536 dimensions, and Voyage 4 supports 256-2048. (OpenAI Platform)

For one million float32 vectors, the raw storage difference is material:

These figures exclude the vector index, metadata, replicas and backups.

The correct question is not:

What is the largest dimension available?

It is:

At what dimension does additional retrieval quality stop justifying additional infrastructure cost?

Test at least two supported dimensions before committing to the index schema.

Context length and chunking strategy

A large context limit is useful, but it does not eliminate document design.

Cohere Embed v4 supports a 128K context, Voyage 4 supports 32K, and OpenAI's current third-generation embedding models accept up to 8192 tokens. (Cohere Documentation)

That does not mean an entire 100-page report should become one vector.

A useful chunk must preserve enough context to answer the question while remaining narrow enough to represent one coherent subject.

Your test should therefore evaluate model and chunking combinations:

A weaker model with an effective chunking and reranking strategy can outperform a stronger model in a poor pipeline.

Deployment, governance and licensing

The model shortlist should change when the system handles: confidential client data; regulated records; intellectual property; jurisdiction-restricted information; documents with granular user permissions; and workloads that must remain on-premises.

A managed API reduces model-serving work but introduces a vendor, data-transfer and availability dependency.

An open-weight model increases control but transfers responsibility for infrastructure; scaling; patching; monitoring; model serving; security; licence compliance; and performance optimisation.

Open weights also do not guarantee commercial permission. Jina Embeddings v5 Text Small, for example, is published under CC BY-NC 4.0, while BGE-M3 uses MIT and Qwen3 Embedding uses Apache 2.0. (Hugging Face)

Licence review must happen before performance testing turns into production adoption.

Migration and vendor resilience

Embedding spaces are model-specific. Vectors generated by unrelated models cannot normally be compared inside the same index.

Switching models may require: re-embedding the complete corpus; creating a second index; synchronising new and updated documents to both indexes; running A/B retrieval tests; migrating traffic; and retiring the old index.

The larger and more frequently changing the corpus, the more important this migration cost becomes.

Voyage's fourth-generation model family is unusual because its large, standard and lite variants produce compatible embeddings, which can reduce some model-tier migration friction inside that family. (Voyage AI)

Compatibility inside one vendor family still does not remove broader vendor dependency. Production architecture should preserve: original source documents; deterministic chunk IDs; model and dimension metadata; re-embedding queues; versioned indexes; and rollback capability.

A decision score for embedding models

A model should be scored as part of the full operating system, not in isolation.

The following default weighting can be adapted to the business case:

A model that wins a public benchmark but fails a mandatory data-residency requirement receives a failing score, not a small penalty.

Similarly, a low-cost model that reduces evidence retrieval below the business acceptance threshold is not economical. It merely moves cost from infrastructure to incorrect answers, manual checking and user distrust.

How to run the evaluation

Define the operating problem

Document who will use the system; which decisions it supports; source types; languages; corpus size and growth; acceptable response time; data sensitivity; cost ceiling; and consequences of incorrect retrieval.

Without this step, the team is evaluating technology without defining success.

Build a representative gold set

Create approximately 100-300 validated questions for an initial production-oriented evaluation.

Each question should contain expected source document; expected evidence passage; acceptable alternative passages; query category; language; difficulty; and whether the correct result is "no evidence found."

Synthetic questions can help expand the set, but subject-matter experts should verify the critical examples.

Test a deliberately small shortlist

A reasonable shortlist might contain one lower-cost managed model; one higher-quality managed model; one multilingual option; one commercially permitted open-weight model; and one hybrid retrieval configuration.

Current examples include OpenAI text-embedding-3-small and text-embedding-3-large, Voyage 4, Cohere Embed v4, BGE-M3, Qwen3 Embedding and Snowflake Arctic Embed 2.0. Their official documentation shows materially different dimensions, context windows, modalities, licences and deployment characteristics. (OpenAI Platform)

The shortlist should remain small enough to evaluate properly.

Measure quality and operating cost

At minimum, record:

Do not combine all results into one average before inspecting failure slices. Ten failures involving obscure internal news may be acceptable. Ten failures involving contractual exclusions may not be.

Run an end-to-end acceptance test

Retrieval metrics are necessary but not sufficient.

Connect the selected retrieval configurations to the intended: vector database; filters; reranker; prompt; LLM; and access-control layer.

Then evaluate whether the final response: uses the correct evidence; attributes it accurately; distinguishes fact from inference; refuses unsupported conclusions; preserves document-level permissions; and meets the required latency.

Only then should the team approve full-corpus indexing.

Which model class fits which situation?

Cohere Embed v4 supports text, images and mixed PDF content. BGE-M3 can produce dense, sparse and multi-vector representations in one model. These capabilities are valuable only when the corresponding retrieval problem exists. (Cohere Documentation)

Where selection projects go wrong

Selecting the highest leaderboard score

A benchmark winner may fail on internal acronyms, identifiers or multilingual queries.

Selecting by API price alone

Embedding generation may be cheap relative to vector storage, model serving, human review and future re-indexing.

Selecting by context window

Long input support does not prove precise retrieval from long documents.

Assuming open source means free operations

The API invoice disappears, but GPU capacity, deployment, monitoring, upgrades and engineering labour remain.

Testing models with different pipelines

Changing chunk size, reranker or top-K together with the model makes it impossible to identify the cause of improvement.

Ignoring the no-answer case

A system that always retrieves something will eventually present irrelevant evidence as support.

Treating access control as a later feature

Document and chunk permissions must be designed before indexing. Otherwise the vector store can become a second data repository whose access rules diverge from the source systems.

What to decide before selecting a model

Do not begin with:

Which embedding model should we use?

Begin with:

What evidence must this system retrieve, for whom, under which constraints, and what happens when retrieval is wrong?

Once those questions are answered, the model shortlist becomes smaller and the evaluation becomes measurable.

The costliest embedding decision is rarely choosing the second-best public model. It is indexing a large corpus before defining the acceptance criteria, migration path and operating constraints.

Through AI & Process Automation consulting, Fill System evaluates retrieval requirements, data sources, security constraints, and expected business value before an embedding architecture is committed — so the model decision is grounded in your operating reality, not a leaderboard.

Request a free diagnostic to validate the retrieval decision before paying to embed and maintain the wrong system.