feat(adv-query): v2 - complete Advanced Query Processing for 100% feature completion

[BjornMelin]

Complete Advanced Query Processing Implementation

Related Issue: Completes remaining 20-25% of functionality from closed Issue #87

Priority: Medium
Epic: V2 Query Processing Enhancement
Estimated Effort: 3-4 days

🎯 Objective

Complete the remaining advanced query processing features to achieve 100% implementation of the advanced query processing system, building on the solid foundation already in place.

📋 Current Status Assessment

Based on comprehensive code review of Issue #87:

✅ Already Complete (75-80%)

• Multi-stage search with Query API (production-ready)
• Complete HyDE implementation with caching
• Enhanced Query API usage with prefetch patterns
• Advanced MCP tools and request models
• Comprehensive test coverage for implemented features

🔄 Partially Implemented (15-20%)

• Basic query understanding (4 categories: technical, code, tutorial, general)
• Infrastructure for Matryoshka embeddings (dimension reduction not implemented)

❌ Missing Features (5-10%)

• Advanced query intent classification beyond basic categories
• True Matryoshka embeddings with dimension reduction (512, 768, 1536)
• Centralized query processing pipeline

🔧 Implementation Tasks

1. Advanced Query Intent Classification

Goal: Expand beyond basic 4-category classification to comprehensive query understanding

Implementation:

# Enhanced query understanding in src/services/hyde/generator.py
class QueryIntentClassifier:
    def classify_query_intent(self, query: str) -> dict:
        """Enhanced query classification with confidence scores"""
        return {
            "primary_intent": str,  # search, code_help, tutorial, troubleshooting
            "secondary_intents": list[str],
            "confidence_score": float,
            "domain_hints": list[str],
            "complexity_level": str,  # beginner, intermediate, advanced
            "expected_format": str,  # code, docs, examples, reference
        }

Files to Modify:

• src/services/hyde/generator.py - Enhance _classify_query() method
• src/services/hyde/config.py - Add intent classification configuration
• tests/unit/services/test_hyde_generator.py - Add comprehensive intent classification tests

Acceptance Criteria:

• Support 10+ intent categories (vs current 4)
• Confidence scoring for classification decisions
• Multi-intent detection (query can have multiple intents)
• Domain-specific hint generation
• Complexity level assessment
• Test coverage >90% for new classification logic

2. True Matryoshka Embeddings Implementation

Goal: Implement dimension reduction capabilities for OpenAI embeddings

Implementation:

# Enhanced embedding manager in src/services/embeddings/manager.py
async def generate_matryoshka_embeddings(
    self, 
    texts: list[str], 
    dimensions: int = 1536
) -> dict:
    """Generate embeddings with specific dimension reduction"""
    # Support for 512, 768, 1536 dimensions
    # Multi-stage retrieval with increasing dimensions
    # Cost/accuracy tradeoff optimization

Files to Modify:

• src/services/embeddings/manager.py - Add Matryoshka support
• src/services/core/qdrant_service.py - Update multi-stage search for dimension awareness
• src/config/enums.py - Add embedding dimension configurations
• tests/unit/services/test_embedding_providers.py - Add Matryoshka tests

Acceptance Criteria:

• Support OpenAI dimension reduction (512, 768, 1536)
• Multi-stage retrieval with increasing dimensions
• Cost/accuracy tradeoff optimization
• Dimension-aware caching strategies
• Fallback handling for dimension mismatches
• Performance benchmarks for different configurations

3. Centralized Query Processing Pipeline

Goal: Create unified query processing orchestration

Implementation:

# New centralized pipeline in src/services/query/pipeline.py
class QueryProcessingPipeline:
    async def process_query(
        self, 
        query: str, 
        collection: str,
        processing_options: dict
    ) -> dict:
        """Centralized query processing with all enhancement stages"""
        # 1. Query intent classification
        # 2. Query enhancement/expansion  
        # 3. Embedding generation (with Matryoshka if needed)
        # 4. Multi-stage search execution
        # 5. Result post-processing and ranking

Files to Create:

• src/services/query/ - New query processing package
• src/services/query/pipeline.py - Central orchestration
• src/services/query/enhancer.py - Query enhancement logic
• tests/unit/services/query/ - Comprehensive pipeline tests

Acceptance Criteria:

• Unified interface for all query processing features
• Configurable pipeline stages
• Performance monitoring and metrics
• Error handling with graceful degradation
• Comprehensive integration tests
• Documentation for pipeline usage

4. Enhanced MCP Integration

Goal: Update MCP tools to use new query processing capabilities

Implementation:

• Update existing MCP tools to use centralized pipeline
• Add new MCP tools for advanced query features
• Enhance request/response models

Files to Modify:

• src/mcp/tools/search.py - Integrate pipeline
• src/mcp/models/requests.py - Add new request models
• src/unified_mcp_server.py - Register new tools

Acceptance Criteria:

• All search tools use centralized pipeline
• New tools for intent classification and query enhancement
• Backward compatibility maintained
• MCP tool documentation updated

🧪 Testing Strategy

Unit Tests

• Query intent classification accuracy tests
• Matryoshka embeddings dimension tests
• Pipeline orchestration tests
• Error handling and edge case tests

Integration Tests

• End-to-end query processing workflow tests
• MCP tool integration tests
• Performance regression tests
• Backward compatibility tests

Performance Benchmarks

• Query processing latency measurements
• Matryoshka embedding performance comparison
• Memory usage optimization validation
• Cost analysis for different configurations

📊 Success Metrics

Feature Completeness

• 100% of advanced query processing features implemented
• All partially implemented features completed
• No regression in existing functionality

Performance Targets

• Query intent classification: <10ms latency
• Matryoshka embeddings: 20-30% cost reduction for lower dimensions
• Pipeline processing: <50ms end-to-end latency
• Memory usage: No increase from current baseline

Quality Targets

• Test coverage: >90% for all new code
• Documentation: 100% coverage of new features
• Code quality: Pass all linting and formatting checks
• Error handling: Comprehensive error coverage

🔗 Dependencies

Required

• Existing Query API implementation (already complete)
• HyDE system (already complete)
• MCP server architecture (already complete)
• Embedding manager (already complete)

Optional

• Performance monitoring tools (can be added later)
• Advanced caching strategies (V2 feature)

📚 Documentation Requirements

Implementation Documentation

• Query intent classification guide
• Matryoshka embeddings usage documentation
• Query processing pipeline architecture
• MCP integration examples

User Documentation

• Advanced query processing tutorial
• Performance optimization guide
• Configuration reference
• Troubleshooting guide

🚀 Implementation Plan

Phase 1: Query Intent Classification (1-2 days)

1. Enhance query classification logic
2. Add comprehensive intent categories
3. Implement confidence scoring
4. Create test suite

Phase 2: Matryoshka Embeddings (1-2 days)

1. Add dimension reduction support
2. Update multi-stage search integration
3. Implement cost optimization logic
4. Create performance benchmarks

Phase 3: Centralized Pipeline (1 day)

1. Create pipeline orchestration
2. Integrate all components
3. Add error handling and monitoring
4. Update MCP tools

Phase 4: Testing & Documentation (0.5 days)

1. Comprehensive test suite
2. Performance validation
3. Documentation updates
4. Integration verification

✅ Definition of Done

• All missing features implemented and tested
• 100% feature completion achieved
• No performance regressions
• Comprehensive documentation updated
• All tests passing (>90% coverage)
• MCP integration fully functional
• Code review completed and approved

🔄 Follow-up Tasks

After completion, these related V2 features can be considered:

• Query expansion with synonyms and related terms
• Advanced caching for query processing results
• Machine learning-based query optimization
• Cross-collection federated search
• Real-time query analytics and monitoring

---

Technical Notes:

• Build on existing solid foundation from Issue feat(v2): implement advanced Query Processing with Multi-Stage Retrieval #87
• Maintain backward compatibility with current MCP tools
• Focus on production-ready implementations
• Prioritize performance and accuracy
• Ensure comprehensive error handling

Estimated Timeline: 3-4 days for complete implementation and testing