What Is HyDE? How to Improve RAG with Hypothetical Documents

                     What Is HyDE? How to Improve RAG with Hypothetical Documents
               



Tópico:
                     What Is HyDE? How to Improve RAG with Hypothetical Documents
               
Categoria: Tutoriais | FreeCodeCamp Premium
Idioma Principal: Português (Conteúdo de Tecnologia)

Conteúdo do Tutorial / Guia Passo a Passo:
-------------------------------------------------------------------------
Retrieval-Augmented Generation, commonly known as RAG, has become one of the most widely used approaches for building applications with large language models.

Instead of asking an LLM to answer entirely from its training data, a RAG system retrieves relevant information from an external knowledge base and provides that information to the model as context.

The basic idea is straightforward:

• Convert the user's question into an embedding.

• Search a vector database for semantically similar document chunks.

• Pass the retrieved chunks to an LLM.

• Generate an answer grounded in those chunks.

But this apparently simple process has a major weakness: the user's question and the document containing the answer may be written very differently.

A user might ask:

Why does my AWS Glue job become significantly slower after processing several million records?

The relevant document in the knowledge base might say:

Performance degradation can occur when Spark executors experience excessive shuffle operations, skewed partitions, memory pressure, or repeated spilling to disk.

The query and the document discuss the same problem, but they use different vocabulary, structure, and levels of detail. A direct query embedding may therefore fail to place them close enough in the embedding space.

This is the problem that Hypothetical Document Embeddings, or HyDE, was designed to solve.

Table of Contents

• Prerequisites

• What is HyDE?

• The Mechanics of HyDE

• Minimal Implementation

• Why Hallucination Doesn't Automatically Break HyDE

• Production Guardrails

• Summary

Prerequisites

To get the most out of this article, there are a few things you should know and have.

What you need to know:

• Basic familiarity with RAG and why it's used.

• How vector embeddings work, at a conceptual level.

• Working knowledge of Python.

What you need to have:

• A local Python environment with numpy, sentence-transformers, and Anthropic installed

• An Anthropic API key if you want to run the HyDE code sample (available at console.anthropic.com)

What is HyDE?

HyDE stands for Hypothetical Document Embeddings. The technique is simple. At query time, you prompt an LLM to generate a hypothetical document that would answer the user's question, embed that document instead of the query, and use its vector to search your index. That's the whole idea. Everything else is engineering.

Figure 2: The HyDE process

The hypothetical document isn't treated as the final answer. It's used only as a bridge between the user's query and the real documents stored in the knowledge base.

This distinction is critical.

The generated document may contain incorrect details. That's not necessarily a failure, because the system doesn't present it directly to the user. Its purpose is to produce a richer semantic representation of the information being sought.

The original HyDE approach used a language model to generate hypothetical documents and an unsupervised dense retriever to map those documents into an embedding space. The embedding acts as a search instruction for retrieving real documents from the corpus.

Why HyDE Works

The intuition is geometric. A dense retriever projects text into a semantic space, and similarity between two pieces of text is the cosine of the angle between their vectors.

When you embed a question and compare it to a passage, you're measuring an angle between two shap

... [O tutorial continua no link abaixo] ...