feat: add documentation for dynamic few-shot examples

Author: Daniel Lorch

docs/few-shot-examples.md

@@ -849,16 +849,274 @@ extraction:
     {DOCUMENT_TEXT}
 ```
 
+## Dynamic Few-Shot Prompting
+
+Dynamic few-shot prompting takes the few-shot examples feature to the next level by automatically selecting the most relevant examples based on document similarity using vector search. Instead of using static examples defined in your configuration, the system dynamically retrieves similar documents from a vector database at processing time.
+
+### Overview
+
+Dynamic few-shot prompting uses Amazon S3 Vectors and Amazon Nova multimodal embeddings to:
+- **Automatically find similar examples** based on visual and content similarity
+- **Retrieve relevant examples** from a large dataset without manual configuration
+- **Adapt to document variations** by selecting the most appropriate examples for each document
+- **Scale to large example datasets** without impacting configuration complexity
+
+### How It Works
+
+```mermaid
+flowchart TD
+    A[Document Processing] --> B[Generate Document Embedding]
+    B --> C[Query S3 Vectors Index]
+    C --> D[Retrieve Top-K Similar Examples]
+    D --> E[Load Example Images from S3]
+    E --> F[Format Examples for Prompt]
+    F --> G[Use in Extraction]
+    
+    subgraph "S3 Vectors"
+        H[Vector Index]
+        I[Example Metadata]
+        J[Similarity Search]
+    end
+    
+    C --> H
+    H --> J
+    J --> I
+    I --> D
+```
+
+### Key Benefits
+
+- **🎯 Automatic Relevance**: Examples are selected based on actual similarity to the document being processed
+- **📈 Scalability**: Support for large example datasets (hundreds or thousands of examples)
+- **🔄 Dynamic Adaptation**: Different examples are selected for different document variations
+- **🚀 No Configuration Overhead**: Add examples to the dataset without modifying configuration files
+- **💡 Better Accuracy**: More relevant examples lead to better extraction results
+- **🔍 Multimodal Similarity**: Uses both visual and content features for similarity matching
+
+### Architecture Components
+
+The dynamic few-shot system consists of:
+
+1. **S3 Vectors Index**: Stores document embeddings and metadata
+2. **Example Dataset**: Collection of example documents with ground truth annotations
+3. **Dynamic Few-Shot Lambda**: Retrieves similar examples based on document embeddings
+4. **IDP Integration**: Seamlessly integrates with Pattern-2 extraction process
+
+### Setting Up Dynamic Few-Shot
+
+#### Step 1: Deploy the Dynamic Few-Shot Stack
+
+Deploy the Lambda function and S3 Vectors infrastructure:
+
+```bash
+cd notebooks/examples/dynamic-few-shot-lambda
+sam deploy --guided
+```
+
+This creates:
+- Lambda function for dynamic example retrieval
+- S3 Vectors bucket and index
+- S3 bucket for storing example images
+- Required IAM roles and permissions
+
+#### Step 2: Populate the Example Dataset
+
+Use the provided notebook to import example documents into S3 Vectors:
+
+```bash
+# Open the dataset import notebook
+jupyter notebook notebooks/misc/fewshot_dataset_import.ipynb
+```
+
+The notebook demonstrates:
+- Loading example documents (e.g., FATURA2 invoice dataset)
+- Generating embeddings using Amazon Nova
+- Uploading vectors and metadata to S3 Vectors
+- Verifying the import with similarity search
+
+**Example Dataset Structure**:
+
+Each example in S3 Vectors includes:
+- **Vector**: Multimodal embedding of the document image
+- **Metadata**:
+  - `classLabel`: Document class (e.g., "invoice")
+  - `classPrompt`: Description for classification
+  - `attributesPrompt`: Expected attribute extraction results in JSON format
+  - `imagePath`: S3 path to example document images
+
+```python
+# Example metadata structure
+metadata = {
+    "classLabel": "invoice",
+    "classPrompt": "This is an example of the class 'invoice'",
+    "attributesPrompt": """expected attributes are:
+        "invoice_number": "INV-2024-001",
+        "invoice_date": "January 15, 2024",
+        "total_amount": "$1,250.00"
+    """,
+    "imagePath": "s3://examples-bucket/invoices/example-001/"
+}
+```
+
+#### Step 3: Configure IDP to Use Dynamic Few-Shot
+
+Add the Lambda ARN to your Pattern-2 extraction configuration:
+
+```yaml
+extraction:
+  dynamic_few_shot_lambda_arn: "arn:aws:lambda:region:account:function:GENAIIDP-dynamic-few-shot"
+```
+
+### Lambda Interface
+
+The dynamic few-shot Lambda receives document information and returns similar examples:
+
+**Input**:
+```json
+{
+  "class_label": "invoice",
+  "document_text": "Text content from the document...",
+  "image_content": ["base64_encoded_image_1", "base64_encoded_image_2"]
+}
+```
+
+**Output**:
+```json
+[
+  {
+    "attributes_prompt": "expected attributes are: \"invoice_number\": \"INV-001\"...",
+    "class_prompt": "This is an example of the class 'invoice'",
+    "distance": 0.122,
+    "image_content": ["base64_encoded_example_image_1", "base64_encoded_example_image_2"]
+  }
+]
+```
+
+### Configuration Parameters
+
+Configure dynamic few-shot behavior through environment variables in the Lambda:
+
+- **`S3VECTOR_BUCKET`**: S3 Vectors bucket name
+- **`S3VECTOR_INDEX`**: Vector index name
+- **`S3VECTOR_DIMENSIONS`**: Embedding dimensions (3072 for Nova Multimodal)
+- **`MODEL_ID`**: Bedrock model for embeddings (e.g., `amazon.nova-2-multimodal-embeddings-v1:0`)
+- **`TOP_K`**: Number of similar examples to retrieve (default: 3)
+- **`THRESHOLD`**: Maximum distance threshold for filtering results (default: 0.2)
+
+### Best Practices
+
+#### Example Dataset Quality
+
+1. **Diverse Coverage**: Include examples covering different document variations
+2. **High-Quality Annotations**: Ensure ground truth attributes are accurate
+3. **Representative Samples**: Choose examples that represent typical documents
+4. **Regular Updates**: Keep the dataset current with new document types
+
+#### Performance Optimization
+
+1. **Appropriate TOP_K**: Use 2-5 examples for optimal balance between accuracy and cost
+2. **Distance Threshold**: Tune threshold to filter out dissimilar examples
+3. **Batch Processing**: Process similar documents together for better cache utilization
+4. **Monitor Costs**: Track embedding generation and Lambda invocation costs
+
+### Monitoring and Troubleshooting
+
+#### CloudWatch Metrics
+
+Monitor these key metrics:
+- **Lambda Duration**: Time to retrieve and process examples
+- **S3 Vectors Query Time**: Vector similarity search performance
+- **Example Count**: Number of examples returned per request
+- **Error Rate**: Failed example retrievals
+
+#### Common Issues
+
+**No Examples Retrieved**:
+- Check that the vector index contains examples for the document class
+- Verify distance threshold isn't too strict
+- Ensure embeddings are generated correctly
+
+**Poor Example Quality**:
+- Review similarity scores (distances) in Lambda logs
+- Adjust TOP_K to retrieve more/fewer examples
+- Improve example dataset quality and diversity
+
+**High Costs**:
+- Reduce TOP_K to retrieve fewer examples
+- Optimize embedding generation frequency
+- Use appropriate Lambda memory allocation
+
+### Comparison: Static vs Dynamic Few-Shot
+
+| Aspect | Static Few-Shot | Dynamic Few-Shot |
+|--------|----------------|------------------|
+| **Configuration** | Examples in YAML config | Lambda ARN in config |
+| **Example Selection** | Fixed examples per class | Similarity-based selection |
+| **Scalability** | Limited by config size | Scales to large datasets |
+| **Relevance** | Same examples for all docs | Most relevant examples per doc |
+| **Setup Complexity** | Simple YAML configuration | Requires Lambda + S3 Vectors |
+| **Cost** | Token cost for examples | Token + Lambda + embedding costs |
+| **Use Case** | Small, stable example sets | Large, diverse example datasets |
+| **Maintenance** | Manual config updates | Add examples to dataset |
+
+### When to Use Dynamic Few-Shot
+
+Consider dynamic few-shot prompting when:
+- You have a large collection of example documents (100+)
+- Document variations are significant within each class
+- You want to minimize configuration maintenance
+- Accuracy improvements justify the additional infrastructure
+- You need to continuously add new examples without config changes
+
+### Example Workflow
+
+1. **Initial Setup**: Deploy Lambda and S3 Vectors infrastructure
+2. **Dataset Import**: Import example documents using the provided notebook
+3. **Configuration**: Add Lambda ARN to IDP extraction config
+4. **Processing**: IDP automatically retrieves relevant examples for each document
+5. **Monitoring**: Track accuracy improvements and costs
+6. **Iteration**: Add more examples to improve coverage
+
+### Cost Considerations
+
+Dynamic few-shot adds these costs:
+- **Embedding Generation**: Nova multimodal embedding API calls
+- **Lambda Invocations**: Per-document Lambda execution
+- **S3 Vectors**: Storage and query costs
+- **S3 Storage**: Example image storage
+
+**Cost Optimization**:
+- Use appropriate TOP_K values (2-3 examples typically sufficient)
+- Implement distance thresholds to avoid irrelevant examples
+- Monitor and optimize Lambda memory allocation
+- Use S3 lifecycle policies for example image storage
+
+### Testing Dynamic Few-Shot
+
+Test your dynamic few-shot setup using the provided notebook:
+
+```bash
+# Open the test notebook
+jupyter notebook notebooks/examples/step3_extraction_with_dynamic_few_shot.ipynb
+```
+
+The notebook demonstrates:
+- Document processing with dynamic few-shot enabled
+- Comparison with static few-shot and no examples
+- Accuracy and cost analysis
+- Example retrieval visualization
+
 ## Future Capabilities
 
 The few-shot examples feature continues to evolve with planned enhancements:
 
-- **Dynamic Example Selection**: Automatically choose the most relevant examples based on document similarity
+- **Hybrid Example Selection**: Combine static and dynamic examples for optimal results
 - **Quality Assessment**: Automated evaluation of example quality and recommendations for improvement
 - **Confidence Integration**: Use few-shot examples to improve confidence scoring
 - **Additional Formats**: Support for more example formats and metadata
 - **Enhanced Caching**: More sophisticated caching strategies for different use cases
-- **Hybrid Examples**: Better integration of text and image content within examples
-- **Enhanced Multi-Image Support**: Advanced algorithms for optimal image selection from directories
+- **Multi-Modal Fusion**: Better integration of text and image content within examples
+- **Adaptive Thresholds**: Automatically adjust similarity thresholds based on document characteristics
+- **Example Ranking**: Advanced algorithms for optimal example selection and ordering
 
-Few-shot examples with prompt caching represent a significant step forward in making Pattern-2 more accurate, cost-effective, and easier to configure for diverse document processing scenarios. The flexibility to use images, text, or both, combined with support for multiple images per example, provides comprehensive options for different security, privacy, and performance requirements.
+Few-shot examples with prompt caching and dynamic selection represent a significant step forward in making Pattern-2 more accurate, cost-effective, and easier to configure for diverse document processing scenarios. The flexibility to use static or dynamic examples, images or text, combined with support for multiple images per example, provides comprehensive options for different security, privacy, and performance requirements.