RAG Query Understanding

Comprehensive query understanding system for enterprise RAG, providing intent classification, query expansion, rewriting, and context-aware processing capabilities.

Core Classes

QueryUnderstandingEngine

Description: Main engine for comprehensive query understanding and processing

Parameters:

• intent_classifier (IntentClassifier): Intent classification component
• query_expander (QueryExpander): Query expansion component
• query_rewriter (QueryRewriter): Query rewriting component
• context_manager (ContextManager): Context management component
• domain_processor (DomainProcessor): Domain-specific processing

Returns: QueryUnderstandingEngine instance

Example:

from recoagent.rag.query_understanding import QueryUnderstandingEngine, IntentClassifier, QueryExpander

# Create components
intent_classifier = IntentClassifier(model_name="intent-classification-model")
query_expander = QueryExpander(domain_terms="medical_terms.json")
query_rewriter = QueryRewriter(rewriting_rules="rewriting_rules.yaml")

# Create understanding engine
engine = QueryUnderstandingEngine(
    intent_classifier=intent_classifier,
    query_expander=query_expander,
    query_rewriter=query_rewriter
)

# Process query
result = engine.process_query(
    query="How do I treat hypertension?",
    context={"user_domain": "medical", "conversation_history": []}
)

IntentClassifier

Description: Classifies query intents (factual, procedural, comparative, exploratory)

Parameters:

• model_name (str): HuggingFace model name for classification
• confidence_threshold (float): Minimum confidence for classification (default: 0.7)
• enable_multi_intent (bool): Enable multi-intent detection (default: True)

Returns: IntentClassifier instance

Example:

from recoagent.rag.query_understanding import IntentClassifier

# Create intent classifier
classifier = IntentClassifier(
    model_name="microsoft/DialoGPT-medium",
    confidence_threshold=0.8,
    enable_multi_intent=True
)

# Classify query intent
intent_result = classifier.classify("What is machine learning and how does it work?")
print(f"Primary Intent: {intent_result.primary_intent}")
print(f"Confidence: {intent_result.confidence}")
print(f"Multi-intents: {intent_result.multi_intents}")

QueryExpander

Description: Expands queries using synonyms, domain terminology, and semantic relationships

Parameters:

• domain_terms (str): Path to domain-specific terms file
• synonym_threshold (float): Similarity threshold for synonyms (default: 0.8)
• max_expansions (int): Maximum number of expansions (default: 5)
• enable_semantic_expansion (bool): Enable semantic expansion (default: True)

Returns: QueryExpander instance

Example:

from recoagent.rag.query_understanding import QueryExpander

# Create query expander
expander = QueryExpander(
    domain_terms="medical_terminology.json",
    synonym_threshold=0.85,
    max_expansions=3,
    enable_semantic_expansion=True
)

# Expand query
expanded_query = expander.expand(
    query="heart disease treatment",
    domain="medical"
)

print(f"Original: {expanded_query.original_query}")
print(f"Expanded: {expanded_query.expanded_queries}")
print(f"Expansion terms: {expanded_query.expansion_terms}")

QueryRewriter

Description: Rewrites queries for better retrieval performance

Parameters:

• rewriting_rules (str): Path to rewriting rules file
• enable_llm_rewriting (bool): Enable LLM-based rewriting (default: True)
• rewriting_model (str): LLM model for rewriting (default: "gpt-3.5-turbo")

Returns: QueryRewriter instance

Example:

from recoagent.rag.query_understanding import QueryRewriter

# Create query rewriter
rewriter = QueryRewriter(
    rewriting_rules="query_rewriting_rules.yaml",
    enable_llm_rewriting=True,
    rewriting_model="gpt-4"
)

# Rewrite query
rewritten_query = rewriter.rewrite(
    query="I need help with my computer",
    context={"domain": "technical_support"}
)

print(f"Original: {rewritten_query.original_query}")
print(f"Rewritten: {rewritten_query.rewritten_query}")
print(f"Rewriting method: {rewritten_query.method}")

ContextManager

Description: Manages conversation context and history for context-aware processing

Parameters:

• max_context_length (int): Maximum context length (default: 10)
• context_decay (float): Context decay factor (default: 0.9)
• enable_entity_tracking (bool): Enable entity tracking (default: True)

Returns: ContextManager instance

Example:

from recoagent.rag.query_understanding import ContextManager

# Create context manager
context_manager = ContextManager(
    max_context_length=5,
    context_decay=0.8,
    enable_entity_tracking=True
)

# Add context
context_manager.add_turn(
    query="What is diabetes?",
    response="Diabetes is a chronic condition...",
    entities=["diabetes", "chronic condition"]
)

# Get context for current query
context = context_manager.get_context("How is it treated?")
print(f"Context: {context.conversation_history}")
print(f"Entities: {context.tracked_entities}")

Usage Examples

Basic Query Understanding

from recoagent.rag.query_understanding import QueryUnderstandingEngine

# Create understanding engine
engine = QueryUnderstandingEngine()

# Process different types of queries
queries = [
    "What is artificial intelligence?",
    "How do I implement a neural network?",
    "Compare machine learning vs deep learning",
    "Find research papers on computer vision"
]

for query in queries:
    result = engine.process_query(query)
    
    print(f"Query: {query}")
    print(f"Intent: {result.intent.primary_intent}")
    print(f"Confidence: {result.intent.confidence:.3f}")
    print(f"Expanded: {result.expansion.expanded_queries}")
    print(f"Rewritten: {result.rewriting.rewritten_query}")
    print("---")

Advanced Intent Classification

from recoagent.rag.query_understanding import IntentClassifier

# Create advanced intent classifier
classifier = IntentClassifier(
    model_name="microsoft/DialoGPT-medium",
    confidence_threshold=0.7,
    enable_multi_intent=True
)

# Classify complex queries
complex_queries = [
    "What is machine learning and how can I use it for my business?",
    "Compare the pros and cons of different AI frameworks",
    "I need to understand both the theory and practical implementation of deep learning"
]

for query in complex_queries:
    result = classifier.classify(query)
    
    print(f"Query: {query}")
    print(f"Primary Intent: {result.primary_intent}")
    print(f"Confidence: {result.confidence:.3f}")
    
    if result.multi_intents:
        print("Multi-intents detected:")
        for intent, confidence in result.multi_intents.items():
            print(f"  - {intent}: {confidence:.3f}")
    
    print("---")

Domain-Specific Query Expansion

from recoagent.rag.query_understanding import QueryExpander

# Create domain-specific expanders
medical_expander = QueryExpander(
    domain_terms="medical_terminology.json",
    synonym_threshold=0.8,
    max_expansions=5
)

tech_expander = QueryExpander(
    domain_terms="tech_terminology.json",
    synonym_threshold=0.85,
    max_expansions=3
)

# Expand queries in different domains
medical_query = "heart attack symptoms"
tech_query = "machine learning algorithms"

medical_expansion = medical_expander.expand(medical_query, domain="medical")
tech_expansion = tech_expander.expand(tech_query, domain="technology")

print("Medical Query Expansion:")
print(f"Original: {medical_expansion.original_query}")
print(f"Expanded: {medical_expansion.expanded_queries}")
print(f"Domain terms: {medical_expansion.domain_terms}")

print("\nTech Query Expansion:")
print(f"Original: {tech_expansion.original_query}")
print(f"Expanded: {tech_expansion.expanded_queries}")
print(f"Domain terms: {tech_expansion.domain_terms}")

Context-Aware Query Processing

from recoagent.rag.query_understanding import QueryUnderstandingEngine, ContextManager

# Create context-aware engine
context_manager = ContextManager(
    max_context_length=3,
    context_decay=0.9,
    enable_entity_tracking=True
)

engine = QueryUnderstandingEngine(context_manager=context_manager)

# Simulate conversation
conversation = [
    "What is diabetes?",
    "How is it diagnosed?",
    "What are the treatment options?",
    "Are there any side effects?"
]

context = None
for i, query in enumerate(conversation):
    # Process query with context
    result = engine.process_query(query, context=context)
    
    print(f"Turn {i+1}: {query}")
    print(f"Intent: {result.intent.primary_intent}")
    print(f"Context entities: {result.context.tracked_entities}")
    print(f"Rewritten with context: {result.rewriting.rewritten_query}")
    
    # Update context
    context = result.context
    print("---")

Multi-Intent Query Decomposition

from recoagent.rag.query_understanding import MultiIntentProcessor

# Create multi-intent processor
processor = MultiIntentProcessor(
    decomposition_model="gpt-4",
    max_subqueries=3
)

# Process multi-intent queries
multi_intent_queries = [
    "What is machine learning and how do I implement it in Python?",
    "Compare TensorFlow and PyTorch, and show me examples of both",
    "Explain neural networks, their types, and provide implementation code"
]

for query in multi_intent_queries:
    decomposition = processor.decompose(query)
    
    print(f"Original Query: {query}")
    print(f"Detected Intents: {decomposition.intents}")
    print("Sub-queries:")
    
    for i, subquery in enumerate(decomposition.subqueries, 1):
        print(f"  {i}. {subquery.query} (Intent: {subquery.intent})")
    
    print("---")

Query Validation and Clarification

from recoagent.rag.query_understanding import QueryValidator, ClarificationGenerator

# Create validator and clarification generator
validator = QueryValidator(
    min_query_length=3,
    max_query_length=500,
    enable_ambiguity_detection=True
)

clarification_generator = ClarificationGenerator(
    clarification_model="gpt-3.5-turbo",
    max_clarifications=3
)

# Validate and generate clarifications
ambiguous_queries = [
    "help",  # Too vague
    "AI",    # Too short
    "machine learning for beginners with no programming experience who want to understand both theory and practical applications",  # Too long
    "python"  # Ambiguous
]

for query in ambiguous_queries:
    validation = validator.validate(query)
    
    print(f"Query: {query}")
    print(f"Valid: {validation.is_valid}")
    print(f"Issues: {validation.issues}")
    
    if not validation.is_valid:
        clarifications = clarification_generator.generate_clarifications(
            query=query,
            issues=validation.issues
        )
        
        print("Clarification questions:")
        for i, clarification in enumerate(clarifications, 1):
            print(f"  {i}. {clarification.question}")
            print(f"     Options: {clarification.options}")
    
    print("---")

Real-time Query Processing Pipeline

from recoagent.rag.query_understanding import QueryUnderstandingEngine
import asyncio

# Create real-time processing engine
engine = QueryUnderstandingEngine(
    enable_real_time=True,
    processing_timeout=2.0  # 2 seconds max
)

async def process_queries_realtime():
    """Process queries in real-time."""
    query_queue = asyncio.Queue()
    
    # Simulate incoming queries
    queries = [
        "What is machine learning?",
        "How do neural networks work?",
        "Compare supervised vs unsupervised learning"
    ]
    
    # Add queries to queue
    for query in queries:
        await query_queue.put(query)
    
    # Process queries
    while not query_queue.empty():
        query = await query_queue.get()
        
        try:
            # Process with timeout
            result = await asyncio.wait_for(
                engine.process_query_async(query),
                timeout=2.0
            )
            
            print(f"Processed: {query}")
            print(f"Intent: {result.intent.primary_intent}")
            print(f"Processing time: {result.processing_time:.3f}s")
            
        except asyncio.TimeoutError:
            print(f"Timeout processing: {query}")
        
        print("---")

# Run real-time processing
asyncio.run(process_queries_realtime())

API Reference

QueryUnderstandingEngine Methods

process_query(query: str, context: Dict = None) -> QueryResult

Process query with full understanding pipeline

Parameters:

• query (str): Input query
• context (Dict, optional): Conversation context

Returns: QueryResult with intent, expansion, and rewriting

process_query_async(query: str, context: Dict = None) -> QueryResult

Async version of query processing

Parameters:

• query (str): Input query
• context (Dict, optional): Conversation context

Returns: QueryResult with processing time

IntentClassifier Methods

classify(query: str) -> IntentResult

Classify query intent

Parameters:

• query (str): Input query

Returns: IntentResult with primary and multi-intents

get_intent_confidence(query: str, intent: str) -> float

Get confidence for specific intent

Parameters:

• query (str): Input query
• intent (str): Intent to check

Returns: Confidence score (0.0-1.0)

QueryExpander Methods

expand(query: str, domain: str = None) -> ExpansionResult

Expand query with synonyms and domain terms

Parameters:

• query (str): Input query
• domain (str, optional): Domain context

Returns: ExpansionResult with expanded queries

add_domain_terms(domain: str, terms: Dict[str, List[str]]) -> None

Add domain-specific terms

Parameters:

• domain (str): Domain name
• terms (Dict): Domain terms and synonyms

QueryRewriter Methods

rewrite(query: str, context: Dict = None) -> RewritingResult

Rewrite query for better retrieval

Parameters:

• query (str): Input query
• context (Dict, optional): Context for rewriting

Returns: RewritingResult with rewritten query

ContextManager Methods

add_turn(query: str, response: str, entities: List[str] = None) -> None

Add conversation turn to context

Parameters:

• query (str): User query
• response (str): System response
• entities (List[str], optional): Extracted entities

get_context(current_query: str = None) -> ContextResult

Get current conversation context

Parameters:

• current_query (str, optional): Current query for context

Returns: ContextResult with conversation history and entities

See Also

• RAG Retrievers - Document retrieval
• RAG Rerankers - Document reranking
• RAG Evaluators - Query evaluation
• Conversational Search Core - Conversational search