RAG Retrievers

Retrieval components for RAG systems, providing various retrieval strategies including hybrid retrieval, vector similarity search, and BM25 keyword matching.

Core Classes

HybridRetriever

Description: Combines multiple retrieval methods for improved results

Parameters:

• vector_retriever (VectorRetriever): Vector similarity retriever
• bm25_retriever (BM25Retriever): BM25 keyword retriever
• fusion_method (str): Fusion method ("rrf", "weighted", "reciprocal")
• vector_weight (float): Weight for vector results (default: 0.7)
• bm25_weight (float): Weight for BM25 results (default: 0.3)

Returns: HybridRetriever instance

Example:

from recoagent.rag.retrievers import HybridRetriever, VectorRetriever, BM25Retriever

# Create individual retrievers
vector_retriever = VectorRetriever(
    vector_store=vector_store,
    embedding_model="text-embedding-ada-002"
)

bm25_retriever = BM25Retriever(k1=1.2, b=0.75)

# Create hybrid retriever
hybrid_retriever = HybridRetriever(
    vector_retriever=vector_retriever,
    bm25_retriever=bm25_retriever,
    fusion_method="rrf",
    vector_weight=0.7,
    bm25_weight=0.3
)

# Retrieve documents
results = hybrid_retriever.retrieve("machine learning algorithms", k=10)

VectorRetriever

Description: Vector similarity-based retriever using embeddings

Parameters:

• vector_store (VectorStore): Vector store instance
• embedding_model (str): Embedding model name
• similarity_threshold (float): Minimum similarity threshold (default: 0.0)
• search_type (str): Search type ("similarity", "mmr")

Returns: VectorRetriever instance

Example:

from recoagent.rag.retrievers import VectorRetriever
from recoagent.rag.stores import OpenSearchStore

# Create vector store
vector_store = OpenSearchStore(
    host="localhost",
    port=9200,
    index_name="documents"
)

# Create vector retriever
vector_retriever = VectorRetriever(
    vector_store=vector_store,
    embedding_model="text-embedding-ada-002",
    similarity_threshold=0.7,
    search_type="similarity"
)

# Retrieve similar documents
results = vector_retriever.retrieve("artificial intelligence", k=5)

BM25Retriever

Description: BM25-based keyword retriever for exact and fuzzy matching

Parameters:

• k1 (float): BM25 k1 parameter (default: 1.2)
• b (float): BM25 b parameter (default: 0.75)
• enable_fuzzy (bool): Enable fuzzy matching (default: False)

Returns: BM25Retriever instance

Example:

from recoagent.rag.retrievers import BM25Retriever
from recoagent.rag.chunkers import Chunk

# Create BM25 retriever
bm25_retriever = BM25Retriever(
    k1=1.2,
    b=0.75,
    enable_fuzzy=True
)

# Add documents
chunks = [
    Chunk(content="Machine learning is a subset of AI", metadata={"source": "doc1"}),
    Chunk(content="Deep learning uses neural networks", metadata={"source": "doc2"})
]
bm25_retriever.add_chunks(chunks)

# Retrieve documents
results = bm25_retriever.retrieve("machine learning", k=5)

RetrievalResult

Description: Represents a retrieval result with score and metadata

Fields:

• chunk (Chunk): Retrieved document chunk
• score (float): Retrieval score
• retrieval_method (str): Method used for retrieval

Usage Examples

Basic Hybrid Retrieval

from recoagent.rag.retrievers import HybridRetriever, VectorRetriever, BM25Retriever
from recoagent.rag.stores import OpenSearchStore

# Setup vector store
vector_store = OpenSearchStore(
    host="localhost",
    port=9200,
    index_name="knowledge_base"
)

# Create retrievers
vector_retriever = VectorRetriever(
    vector_store=vector_store,
    embedding_model="text-embedding-ada-002"
)

bm25_retriever = BM25Retriever(k1=1.2, b=0.75)

# Create hybrid retriever
hybrid_retriever = HybridRetriever(
    vector_retriever=vector_retriever,
    bm25_retriever=bm25_retriever,
    fusion_method="rrf"
)

# Retrieve documents
query = "What are the benefits of machine learning?"
results = hybrid_retriever.retrieve(query, k=10)

# Process results
for result in results:
    print(f"Score: {result.score:.3f}")
    print(f"Method: {result.retrieval_method}")
    print(f"Content: {result.chunk.content[:100]}...")
    print("---")

Advanced Vector Retrieval

from recoagent.rag.retrievers import VectorRetriever
from recoagent.rag.stores import AzureAISearchStore

# Create Azure AI Search store
azure_store = AzureAISearchStore(
    endpoint="https://your-search.search.windows.net",
    api_key="your-api-key",
    index_name="documents"
)

# Create advanced vector retriever
vector_retriever = VectorRetriever(
    vector_store=azure_store,
    embedding_model="text-embedding-ada-002",
    similarity_threshold=0.8,
    search_type="mmr"  # Maximum Marginal Relevance
)

# Retrieve with filters
results = vector_retriever.retrieve(
    query="artificial intelligence applications",
    k=10,
    filters={
        "category": "technology",
        "date_range": "2023-01-01,2024-01-01"
    }
)

# Analyze results
print(f"Retrieved {len(results)} documents")
for result in results:
    print(f"Similarity: {result.score:.3f}")
    print(f"Source: {result.chunk.metadata.get('source', 'Unknown')}")

BM25 with Custom Parameters

from recoagent.rag.retrievers import BM25Retriever
from recoagent.rag.chunkers import Chunk

# Create BM25 retriever with custom parameters
bm25_retriever = BM25Retriever(
    k1=1.5,  # Higher k1 for more term frequency sensitivity
    b=0.8,   # Higher b for more length normalization
    enable_fuzzy=True
)

# Add documents with metadata
documents = [
    {
        "content": "Machine learning algorithms can be supervised or unsupervised",
        "metadata": {"category": "ml", "difficulty": "beginner"}
    },
    {
        "content": "Deep learning is a subset of machine learning using neural networks",
        "metadata": {"category": "dl", "difficulty": "intermediate"}
    },
    {
        "content": "Natural language processing uses machine learning for text analysis",
        "metadata": {"category": "nlp", "difficulty": "intermediate"}
    }
]

# Convert to chunks
chunks = [
    Chunk(content=doc["content"], metadata=doc["metadata"])
    for doc in documents
]

bm25_retriever.add_chunks(chunks)

# Retrieve with different queries
queries = [
    "machine learning algorithms",
    "neural networks deep learning",
    "text analysis NLP"
]

for query in queries:
    results = bm25_retriever.retrieve(query, k=3)
    print(f"\nQuery: {query}")
    for result in results:
        print(f"  Score: {result.score:.3f} - {result.chunk.content[:50]}...")

Multi-Modal Retrieval

from recoagent.rag.retrievers import VectorRetriever
from recoagent.rag.stores import VertexAIVectorStore

# Create multi-modal vector store
vertex_store = VertexAIVectorStore(
    project_id="your-project",
    location="us-central1",
    index_name="multimodal-index"
)

# Create multi-modal retriever
multimodal_retriever = VectorRetriever(
    vector_store=vertex_store,
    embedding_model="multimodal-embedding-001",
    similarity_threshold=0.7
)

# Retrieve with different modalities
text_query = "Show me images of machine learning concepts"
image_query = "path/to/query_image.jpg"

# Text-based retrieval
text_results = multimodal_retriever.retrieve(text_query, k=5)

# Image-based retrieval
image_results = multimodal_retriever.retrieve_by_image(image_query, k=5)

# Combine results
all_results = text_results + image_results

Retrieval with Reranking

from recoagent.rag.retrievers import HybridRetriever
from recoagent.rag.rerankers import CrossEncoderReranker

# Create reranker
reranker = CrossEncoderReranker(
    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2"
)

# Create hybrid retriever
hybrid_retriever = HybridRetriever(
    vector_retriever=vector_retriever,
    bm25_retriever=bm25_retriever,
    fusion_method="weighted"
)

# Retrieve and rerank
query = "What are the latest developments in AI?"
initial_results = hybrid_retriever.retrieve(query, k=20)  # Get more results

# Rerank for better relevance
reranked_results = reranker.rerank(
    query=query,
    documents=[result.chunk.content for result in initial_results],
    top_k=10
)

# Final results with reranking scores
final_results = []
for i, (doc, score) in enumerate(reranked_results):
    original_result = initial_results[i]
    final_results.append(RetrievalResult(
        chunk=original_result.chunk,
        score=score,
        retrieval_method=f"{original_result.retrieval_method}+rerank"
    ))

API Reference

HybridRetriever Methods

retrieve(query: str, k: int = 5) -> List[RetrievalResult]

Retrieve documents using hybrid approach

Parameters:

• query (str): Search query
• k (int): Number of documents to retrieve

Returns: List of retrieval results

set_weights(vector_weight: float, bm25_weight: float) -> None

Update retrieval weights

Parameters:

• vector_weight (float): Weight for vector results
• bm25_weight (float): Weight for BM25 results

VectorRetriever Methods

retrieve(query: str, k: int = 5, filters: Dict = None) -> List[RetrievalResult]

Retrieve documents using vector similarity

Parameters:

• query (str): Search query
• k (int): Number of documents
• filters (Dict, optional): Additional filters

Returns: List of retrieval results

retrieve_by_image(image_path: str, k: int = 5) -> List[RetrievalResult]

Retrieve documents using image similarity (multi-modal)

Parameters:

• image_path (str): Path to query image
• k (int): Number of documents

Returns: List of retrieval results

BM25Retriever Methods

add_chunks(chunks: List[Chunk]) -> None

Add documents to BM25 index

Parameters:

• chunks (List[Chunk]): Document chunks to index

retrieve(query: str, k: int = 5) -> List[RetrievalResult]

Retrieve documents using BM25 scoring

Parameters:

• query (str): Search query
• k (int): Number of documents

Returns: List of retrieval results

update_parameters(k1: float, b: float) -> None

Update BM25 parameters

Parameters:

• k1 (float): BM25 k1 parameter
• b (float): BM25 b parameter

See Also

• RAG Rerankers - Document reranking
• RAG Stores - Vector store backends
• RAG Chunkers - Document chunking
• RAG Evaluators - Retrieval evaluation