Merge branch 'feature/bounding-boxes' into 'develop'

Bounding boxes

See merge request genaiic-reusable-assets/engagement-artifacts/genaiic-idp-accelerator!268

Author: rstrahan

CHANGELOG.md

@@ -7,74 +7,30 @@ SPDX-License-Identifier: MIT-0
 
 ### Added
 
-- **Modern Python-Based Publishing System**
-  - **Complete Rewrite**: Replaced legacy `publish.sh` bash script (583 lines) with modern `publish.py` Python script (1,294 lines)
-  - **Enhanced User Experience**: Rich console interface with progress bars, spinners, and colored output using Rich library
-  - **Concurrent Processing**: Multi-threaded artifact building and uploading for significantly improved performance
-  - **Cross-Platform Support**: Native support for Linux, macOS, and Windows environments
-  - **Intelligent Caching**: Advanced checksum-based build optimization to skip unnecessary rebuilds
-  - **Robust Error Handling**: Comprehensive error handling with detailed error messages and recovery suggestions
-  - **Resource Management**: Automatic cleanup of temporary files and proper resource management
-  - **CLI Interface**: Modern command-line interface using Typer with clear help documentation and parameter validation
-  - **Key Features**:
-    - Parallel S3 uploads with thread-safe progress tracking
-    - CloudFormation template validation and processing
-    - Automatic bucket creation with proper permissions and lifecycle policies
-    - Build artifact optimization with dependency tracking
-    - Comprehensive logging and debugging capabilities
-    - Memory-efficient processing for large artifacts
-
-- **Comprehensive Unit Test Suite for Publishing System**
-  - **Extensive Test Coverage**: 1,621 lines of unit tests covering 95%+ of publish.py functionality
-  - **Testing Framework**: Uses pytest with proper unit test markers (`@pytest.mark.unit`) following project testing standards
-  - **Mock Integration**: Comprehensive mocking of AWS services (S3, CloudFormation) for isolated unit testing without external dependencies
-  - **Cross-Platform Testing**: Tests for Linux, macOS, and Windows-specific functionality and edge cases
-  - **Error Scenario Coverage**: Tests for network failures, permission errors, malformed templates, and concurrent access scenarios
-  - **Performance Testing**: Tests for concurrent operations, memory usage, and resource cleanup
-  - **Key Test Categories**:
-    - IDPPublisher initialization and configuration
-    - S3 operations (bucket creation, uploads, permissions)
-    - CloudFormation template processing and validation
-    - Build system integration and checksum validation
-    - Error handling and recovery mechanisms
-    - Concurrent publishing workflows and thread safety
-
-- **Windows Development Environment Setup**
-  - **Automated Setup Script**: New `scripts/dev_setup.bat` (570 lines) for complete Windows development environment configuration
-  - **Comprehensive Installation**: Automated installation of all required development tools and dependencies
-  - **Tool Installation**:
-    - AWS CLI with proper configuration
-    - Python 3.13 (required for compatibility with latest dependencies)
-    - Node.js for React UI development
-    - Git for version control
-    - Docker for containerized development
-    - AWS SAM CLI for serverless application development
-    - Python dependencies (boto3, numpy 2.3.2, typer, rich)
-  - **Environment Configuration**: Automatic AWS credentials setup and project cloning with dependency installation
-  - **Administrator Privileges**: Proper handling of Windows administrator requirements for system-level installations
-  - **Error Handling**: Comprehensive error checking and user guidance throughout the setup process
+- **Pattern-2 and Pattern-3 Assessment now generate geometry (bounding boxes) for visualization in UI 'Visual Editor' (parity with Pattern-1)**
+  - Added comprehensive spatial localization capabilities to both regular and granular assessment services
+  - **Automatic Processing**: When LLM provides bbox coordinates, automatically converts to UI-compatible (Visual Edit) geometry format without any configuration
+  - **Universal Support**: Works with all attribute types - simple attributes, nested group attributes (e.g., CompanyAddress.State), and list attributes
+  - **Enhanced Prompts**: Updated assessment task prompts with spatial-localization-guidelines requesting bbox coordinates in normalized 0-1000 scale
+  - **Demo Notebooks**: Assessment notebooks now showcase automatic bounding box processing
+
+- **New Python-Based Publishing System**
+  - Replaced `publish.sh` bash script with new `publish.py` Python script
+  - Rich console interface with progress bars, spinners, and colored output using Rich library
+  - Multi-threaded artifact building and uploading for significantly improved performance
+  - Native support for Linux, macOS, and Windows environments
+
+- **Windows Development Environment Setup Guide and Helper Script**
+  - New `scripts/dev_setup.bat` (570 lines) for complete Windows development environment configuration
 
 - **OCR Service Default Image Sizing for Resource Optimization**
   - Implemented automatic default image size limits (951×1268) when no image sizing configuration is provided
   - **Key Benefits**: Reduction in vision model token consumption, prevents OutOfMemory errors during concurrent processing, improves processing speed and reduces bandwidth usage
 
 ### Changed
 
-- **Publishing Workflow Modernization**
-  - Migrated from bash-based to Python-based publishing system for better maintainability and cross-platform support
-  - Improved build performance through intelligent caching and concurrent processing
-  - Enhanced developer experience with rich console output and clear progress indicators
-  - Better error diagnostics and troubleshooting capabilities
-
 - **Reverted to python3.12 runtime to resolve build package dependency problems**
 
-### Technical Improvements
-
-- **Build System Optimization**: Checksum-based incremental builds reduce unnecessary processing time
-- **Memory Management**: Efficient handling of large artifacts and proper resource cleanup
-- **Thread Safety**: Concurrent operations with proper synchronization and error handling
-- **Code Quality**: Comprehensive unit testing ensures reliability and maintainability of the publishing system
-
 
 
 

config_library/pattern-2/bank-statement-sample/config.yaml

@@ -368,6 +368,7 @@ summarization:
   model: us.anthropic.claude-3-7-sonnet-20250219-v1:0
   system_prompt: >-
     You are a document summarization expert who can analyze and summarize documents from various domains including medical, financial, legal, and general business documents. Your task is to create a summary that captures the key information, main points, and important details from the document. Your output must be in valid JSON format. \nSummarization Style: Balanced\\nCreate a balanced summary that provides a moderate level of detail. Include the main points and key supporting information, while maintaining the document's overall structure. Aim for a comprehensive yet concise summary.\n Your output MUST be in valid JSON format with markdown content. You MUST strictly adhere to the output format specified in the instructions.
+
 assessment:
   enabled: true
   image:
@@ -383,130 +384,146 @@ assessment:
   max_tokens: '10000'
   top_k: '5'
   temperature: '0.0'
-  model: us.amazon.nova-pro-v1:0
+  model: us.amazon.nova-lite-v1:0
   system_prompt: >-
-    You are a document analysis assessment expert. Your task is to evaluate the confidence of extraction results by analyzing the source document evidence. Respond only with JSON containing confidence scores for each extracted attribute.
+    You are a document analysis assessment expert. Your role is to evaluate the confidence and accuracy of data extraction results by analyzing them against source documents.
+    
+    Provide accurate confidence scores for each assessment.
+    When bounding boxes are requested, provide precise coordinate locations where information appears in the document.
   task_prompt: >-
     <background>
-
-    You are an expert document analysis assessment system. Your task is to evaluate the confidence of extraction results for a document of class {DOCUMENT_CLASS}.
-
+    You are an expert document analysis assessment system. Your task is to evaluate the confidence of extraction results for a document of class {DOCUMENT_CLASS} and provide precise spatial localization for each field.
     </background>
 
-
     <task>
-
-    Analyze the extraction results against the source document and provide confidence assessments for each extracted attribute. Consider factors such as:
-
-    1. Text clarity and OCR quality in the source regions
-    2. Alignment between extracted values and document content
-    3. Presence of clear evidence supporting the extraction
-    4. Potential ambiguity or uncertainty in the source material
+    Analyze the extraction results against the source document and provide confidence assessments AND bounding box coordinates for each extracted attribute. Consider factors such as:
+    1. Text clarity and OCR quality in the source regions 
+    2. Alignment between extracted values and document content 
+    3. Presence of clear evidence supporting the extraction 
+    4. Potential ambiguity or uncertainty in the source material 
     5. Completeness and accuracy of the extracted information
-
+    6. Precise spatial location of each field in the document
     </task>
 
-
     <assessment-guidelines>
-
-    For each attribute, provide:
-    A confidence score between 0.0 and 1.0 where:
+    For each attribute, provide: 
+    - A confidence score between 0.0 and 1.0 where:
        - 1.0 = Very high confidence, clear and unambiguous evidence
        - 0.8-0.9 = High confidence, strong evidence with minor uncertainty
        - 0.6-0.7 = Medium confidence, reasonable evidence but some ambiguity
        - 0.4-0.5 = Low confidence, weak or unclear evidence
        - 0.0-0.3 = Very low confidence, little to no supporting evidence
-
-    Guidelines:
-    - Base assessments on actual document content and OCR quality
-    - Consider both text-based evidence and visual/layout clues
-    - Account for OCR confidence scores when provided
-    - Be objective and specific in reasoning
+    - A clear explanation of the confidence reasoning
+    - Precise spatial coordinates where the field appears in the document
+
+    Guidelines: 
+    - Base assessments on actual document content and OCR quality 
+    - Consider both text-based evidence and visual/layout clues 
+    - Account for OCR confidence scores when provided 
+    - Be objective and specific in reasoning 
     - If an extraction appears incorrect, score accordingly with explanation
-
+    - Provide tight, accurate bounding boxes around the actual text
     </assessment-guidelines>
 
-    <final-instructions>
+    <spatial-localization-guidelines>
+    For each field, provide bounding box coordinates:
+    - bbox: [x1, y1, x2, y2] coordinates in normalized 0-1000 scale
+    - page: Page number where the field appears (starting from 1)
+    
+    Coordinate system:
+    - Use normalized scale 0-1000 for both x and y axes
+    - x1, y1 = top-left corner of bounding box  
+    - x2, y2 = bottom-right corner of bounding box
+    - Ensure x2 > x1 and y2 > y1
+    - Make bounding boxes tight around the actual text content
+    - If a field spans multiple lines, create a bounding box that encompasses all relevant text
+    </spatial-localization-guidelines>
 
-    Analyze the extraction results against the source document and provide confidence assessments. Return a JSON object with the following structure based on the attribute type:
+    <final-instructions>
+    Analyze the extraction results against the source document and provide confidence assessments with spatial localization. Return a JSON object with the following structure based on the attribute type:
 
-    For SIMPLE attributes:
+    For SIMPLE attributes: 
     {
       "simple_attribute_name": {
         "confidence": 0.85,
+        "bbox": [100, 200, 300, 250],
+        "page": 1
       }
     }
 
-    For GROUP attributes (nested object structure):
+    For GROUP attributes (nested object structure): 
     {
       "group_attribute_name": {
         "sub_attribute_1": {
           "confidence": 0.90,
+          "bbox": [150, 300, 250, 320],
+          "page": 1
         },
         "sub_attribute_2": {
           "confidence": 0.75,
+          "bbox": [150, 325, 280, 345],
+          "page": 1
         }
       }
     }
 
-    For LIST attributes (array of assessed items):
+    For LIST attributes (array of assessed items): 
     {
       "list_attribute_name": [
         {
           "item_attribute_1": {
             "confidence": 0.95,
+            "bbox": [100, 400, 200, 420],
+            "page": 1
           },
           "item_attribute_2": {
             "confidence": 0.88,
+            "bbox": [250, 400, 350, 420],
+            "page": 1
           }
         },
         {
           "item_attribute_1": {
             "confidence": 0.92,
+            "bbox": [100, 425, 200, 445],
+            "page": 1
           },
           "item_attribute_2": {
             "confidence": 0.70,
+            "bbox": [250, 425, 350, 445],
+            "page": 1
           }
         }
       ]
     }
 
-    IMPORTANT: 
-    - For LIST attributes like "Transactions", assess EACH individual item in the list separately
-    - Each transaction should be assessed as a separate object in the array
-    - Do NOT provide aggregate assessments for list items - assess each one individually
-    - Include assessments for ALL attributes present in the extraction results
+    IMPORTANT:  
+    - For LIST attributes like "Transactions", assess EACH individual item in the list separately with individual bounding boxes
+    - Each transaction should be assessed as a separate object in the array with its own spatial coordinates
+    - Do NOT provide aggregate assessments for list items - assess each one individually with precise locations
+    - Include assessments AND bounding boxes for ALL attributes present in the extraction results
     - Match the exact structure of the extracted data
-
+    - Provide page numbers for all bounding boxes (starting from 1)
     </final-instructions>
 
-    <attributes-definitions>
-
-    {ATTRIBUTE_NAMES_AND_DESCRIPTIONS}
-
-    </attributes-definitions>
-
     <<CACHEPOINT>>
 
     <document-image>
-
     {DOCUMENT_IMAGE}
-
     </document-image>
 
-
     <ocr-text-confidence-results>
-
     {OCR_TEXT_CONFIDENCE}
-
     </ocr-text-confidence-results>
 
     <<CACHEPOINT>>
 
-    <extraction-results>
+    <attributes-definitions>
+    {ATTRIBUTE_NAMES_AND_DESCRIPTIONS}
+    </attributes-definitions>
 
+    <extraction-results>
     {EXTRACTION_RESULTS}
-
     </extraction-results>
 
 evaluation: