Merge branch 'aws-solutions-library-samples:main' into main

Author: mbellaga

.gitignore

@@ -19,3 +19,4 @@ __pycache__
 rvl_cdip_*
 notebooks/examples/data
 .idea/
+.dsr/

CHANGELOG.md

@@ -5,6 +5,83 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+### Added
+
+
+
+## [0.3.12]
+
+### Added
+
+- **Custom Prompt Generator Lambda Support for Patterns 2 & 3**
+  - Added `custom_prompt_lambda_arn` configuration field to enable injection of custom business logic into extraction processing
+  - **Key Features**: Lambda interface with all template placeholders (DOCUMENT_TEXT, DOCUMENT_CLASS, ATTRIBUTE_NAMES_AND_DESCRIPTIONS, DOCUMENT_IMAGE), URI-based image handling for JSON serialization, comprehensive error handling with fail-fast behavior, scoped IAM permissions requiring GENAIIDP-* function naming
+  - **Use Cases**: Document type-specific processing rules, integration with external systems for customer configurations, conditional processing based on document content, regulatory compliance and industry-specific requirements
+  - **Demo Resources**: Interactive notebook demonstration (`step3_extraction_with_custom_lambda.ipynb`), SAM deployment template for demo Lambda function, comprehensive documentation and examples in `notebooks/examples/demo-lambda/`
+  - **Benefits**: Custom business logic without core code changes, backward compatible (existing deployments unchanged), robust JSON serialization handling all object types, complete observability with detailed logging
+
+- **Refactored Document Classification Service for Enhanced Boundary Detection**
+  - Consolidated `multimodalPageLevelClassification` and the experimental `multimodalPageBoundaryClassification` (from v0.3.11) into a single enhanced `multimodalPageLevelClassification` method
+  - Implemented BIO-like sequence segmentation with document boundary indicators: "start" (new document) and "continue" (same document)
+  - Automatically segments multi-document packets, even when they contain multiple documents of the same type
+  - Added comprehensive classification guide with method comparisons and best practices
+  - **Benefits**: Simplified codebase with single multimodal classification method, improved handling of complex document packets, maintains backward compatibility
+  - **No Breaking Changes**: Existing configurations work unchanged, no configuration updates required
+
+- **Enhanced A2I Template and Workflow Management**
+  - Enhanced A2I template with improved user interface and clearer instructions for reviewers
+  - Added comprehensive instructions for reviewers in A2I template to guide the review process
+  - Implemented capture of failed review tasks with proper error handling and logging
+  - Added workflow orchestration control to stop processing when reviewer rejects A2I task
+  - Removed automatic A2I task creation when Pattern-1 Bedrock Data Automation (BDA) fails to classify document to appropriate Blueprint
+
+- **Dynamic Cost Calculation for Metering Data**
+  - Added automated unit cost and estimated cost calculation to metering table with new `unit_cost` and `estimated_cost` columns
+  - Dynamic pricing configuration loading from configuration
+  - Enhanced cost analysis capabilities with comprehensive Athena queries for cost tracking, trend analysis, and efficiency metrics
+  - Automatic cost calculation as `estimated_cost = value × unit_cost` for all metering records
+  
+- **Configuration-Based Summarization Control**
+  - Summarization can now be enabled/disabled via configuration file `summarization.enabled` property instead of CloudFormation stack parameter
+  - **Key Benefits**: Runtime control without stack redeployment, zero LLM costs when disabled, simplified state machine architecture, backward compatible defaults
+  - **Implementation**: Always calls SummarizationStep but service skips processing when `enabled: false`
+  - **Cost Optimization**: When disabled, no LLM API calls or S3 operations are performed
+  - **Configuration Example**: Set `summarization.enabled: false` to disable, `enabled: true` to enable (default)
+
+- **Configuration-Based Assessment Control**
+  - Assessment can now be enabled/disabled via configuration file `assessment.enabled` property instead of CloudFormation stack parameter
+  - **Key Benefits**: Runtime control without stack redeployment, zero LLM costs when disabled, simplified state machine architecture, backward compatible defaults
+  - **Implementation**: Always calls AssessmentStep but service skips processing when `enabled: false`
+  - **Cost Optimization**: When disabled, no LLM API calls or S3 operations are performed
+  - **Configuration Example**: Set `assessment.enabled: false` to disable, `enabled: true` to enable (default)
+
+- **New guides for setting up development environments**
+  - EC2-based Linux development environment
+  - MacOS development environment
+
+### Removed
+- **CloudFormation Parameters**: Removed `IsSummarizationEnabled` and `IsAssessmentEnabled` parameters from all pattern templates
+- **Related Conditions**: Removed parameter conditions and state machine definition substitutions for both features
+- **Conditional Logic**: Eliminated complex conditional logic from state machine definitions for summarization and assessment steps
+
+### ⚠️ Breaking Changes
+- **Configuration Migration Required**: When updating a stack that previously had `IsSummarizationEnabled` or `IsAssessmentEnabled` set to `false`, these features will now default to `enabled: true` after the update. To maintain the disabled behavior:
+  1. Update your configuration file to set `summarization.enabled: false` and/or `assessment.enabled: false` as needed
+  2. Save the configuration changes immediately after the stack update
+  3. This ensures continued cost optimization by preventing unexpected LLM API calls
+- **Action Required**: Review your current CloudFormation parameter settings before updating and update your configuration accordingly to preserve existing behavior
+
+### Changed
+- **Updated Python Lambda Runtime to 3.13**
+
+### Fixed
+- **Fixed B615 "Unsafe Hugging Face Hub download without revision pinning" security finding in Pattern-3 fine-tuning module** - Added revision pinning with to prevent supply chain attacks and ensure reproducible deployments
+- **Fixed CloudWatch Log Group Missing Retention regression**
+- **Security: Cross-Site Scripting (XSS) Vulnerability in FileViewer Component** - Fixed high-risk XSS vulnerability in `src/ui/src/components/document-viewer/FileViewer.jsx` where `innerHTML` was used with user-controlled data
+- **Add permissions boundary support to new Lambda function roles introduced in previous releases**
+- **Fixed OutOfMemory Errors in Pattern-2 OCR Lambda for Large High-Resolution Documents**
+  - **Root Cause**: Processing large PDFs with high-resolution images (7469×9623 pixels) caused memory spikes when 20 concurrent workers each held ~101MB images simultaneously, exceeding the 4GB Lambda memory limit
+  - **Optimal Solution**: Refactored image extraction to render directly at target dimensions using PyMuPDF matrix transformations, completely eliminating oversized image creation
 
 ## [0.3.11]
 

README.md

@@ -33,6 +33,7 @@ White-glove customization, deployment, and integration support for production us
 - **Modular, pluggable patterns**: Pre-built processing patterns using state-of-the-art models and AWS services
 - **Advanced Classification**: Support for page-level and holistic document packet classification
 - **Few Shot Example Support**: Improve accuracy through example-based prompting
+- **Custom Business Logic Integration**: Inject custom prompt generation logic via Lambda functions for specialized document processing
 - **High Throughput Processing**: Handles large volumes of documents through intelligent queuing
 - **Built-in Resilience**: Comprehensive error handling, retries, and throttling management
 - **Cost Optimization**: Pay-per-use pricing model with built-in controls

VERSION

@@ -1 +1 @@
-0.3.11
+0.3.12

config_library/pattern-1/lending-package-sample/config.yaml

@@ -5,6 +5,7 @@ notes: Processing configuration in BDA project.
 assessment:
   default_confidence_threshold: '0.8'
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'

config_library/pattern-2/bank-statement-sample/config.yaml

@@ -307,6 +307,7 @@ extraction:
   system_prompt: >-
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
@@ -368,6 +369,7 @@ summarization:
   system_prompt: >-
     You are a document summarization expert who can analyze and summarize documents from various domains including medical, financial, legal, and general business documents. Your task is to create a summary that captures the key information, main points, and important details from the document. Your output must be in valid JSON format. \nSummarization Style: Balanced\\nCreate a balanced summary that provides a moderate level of detail. Include the main points and key supporting information, while maintaining the document's overall structure. Aim for a comprehensive yet concise summary.\n Your output MUST be in valid JSON format with markdown content. You MUST strictly adhere to the output format specified in the instructions.
 assessment:
+  enabled: true
   image:
     target_height: ''
     target_width: ''

config_library/pattern-2/lending-package-sample/config.yaml

@@ -1,3 +1,9 @@
+# SPDX-License-Identifier: MIT-0
+
+notes: Boundary-aware classification example for pattern-2
+
+
+
 # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
 # SPDX-License-Identifier: MIT-0
 
@@ -914,15 +920,26 @@ classes:
         evaluation_method: LLM
         attributeType: group
 classification:
+  classificationMethod: multimodalPageLevelClassification
   image:
     target_height: ''
     target_width: ''
+  model: us.amazon.nova-pro-v1:0
+  temperature: '0.0'
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
+  system_prompt: >-
+    You are a multimodal document classification expert that analyzes business documents using both visual layout and textual content. Your task is to classify single-page documents into predefined categories based on their structural patterns, visual features, and text content. Your output must be valid JSON according to the requested format.
+
+    <variables>
+    <document-ocr-data>: OCR-extracted text content from the document page that provides textual information for classification
+    <document-image>: Visual representation of the document page that provides layout, formatting, and visual structure information
+    <document-types>: List of valid document types with their descriptions that the document must be classified into
+    </variables>
   task_prompt: >-
     <task-description>
-    Analyze the provided document using both its visual layout and textual content to determine its document type. You must classify it into exactly one of the predefined categories.
+    Analyze the provided document using both its visual layout and textual content to determine its document type and whether this page begins a new document or continues the previous one.
     </task-description>
 
     <document-types>
@@ -934,24 +951,16 @@ classification:
     1. Examine the visual layout: headers, logos, formatting, structure, and visual organization
     2. Analyze the textual content: key phrases, terminology, purpose, and information type
     3. Identify distinctive features that match the document type descriptions
-    4. Consider both visual and textual evidence together to determine the best match
-    5. CRITICAL: Only use document types explicitly listed in the <document-types> section
+    4. Decide if this page starts a new document (output "start") or continues the previous document (output "continue")
+    5. Consider both visual and textual evidence together to determine the best match
+    6. CRITICAL: Only use document types explicitly listed in the <document-types> section
     </classification-instructions>
 
-    <reasoning-guidelines>
-    When determining the document type:
-    - First identify the document's primary purpose and function
-    - Note specific visual elements (letterhead, forms, tables, signatures)
-    - Identify key textual indicators (terminology, phrases, structure)
-    - Consider the document's intended audience and use case
-    - Provide specific evidence from both visual and textual analysis
-    </reasoning-guidelines>
-
     <output-format>
-    Return your classification as valid JSON following this exact structure:
     {
       "classification_reason": "Detailed reasoning including specific visual and textual evidence that led to this classification",
-      "class": "exact_document_type_from_list"
+      "class": "exact_document_type_from_list",
+      "document_boundary": "start or continue"
     }
     </output-format>
 
@@ -968,22 +977,10 @@ classification:
     <final-instructions>
     Analyze the document above by:
     1. Applying the <classification-instructions> to examine both visual and textual features
-    2. Following the <reasoning-guidelines> to build your classification rationale
-    3. Selecting ONLY from document types in <document-types>
-    4. Providing clear reasoning with specific evidence before the classification
-    5. Outputting in the exact JSON format specified in <output-format>
+    2. Selecting ONLY from document types in <document-types>
+    3. Providing clear reasoning with specific evidence
+    4. Outputting in the exact JSON format specified in <output-format>
     </final-instructions>
-  temperature: '0.0'
-  model: us.amazon.nova-pro-v1:0
-  system_prompt: >-
-    You are a multimodal document classification expert that analyzes business documents using both visual layout and textual content. Your task is to classify single-page documents into predefined categories based on their structural patterns, visual features, and text content. Your output must be valid JSON according to the requested format.
-
-    <variables>
-    DOCUMENT_TEXT: OCR-extracted text content from the document page that provides textual information for classification
-    DOCUMENT_IMAGE: Visual representation of the document page that provides layout, formatting, and visual structure information
-    CLASS_NAMES_AND_DESCRIPTIONS: List of valid document types with their descriptions that the document must be classified into
-    </variables>
-  classificationMethod: multimodalPageLevelClassification
 extraction:
   image:
     target_width: ''
@@ -1082,6 +1079,7 @@ extraction:
   system_prompt: >-
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
@@ -1143,6 +1141,7 @@ summarization:
   system_prompt: >-
     You are a document summarization expert who can analyze and summarize documents from various domains including medical, financial, legal, and general business documents. Your task is to create a summary that captures the key information, main points, and important details from the document. Your output must be in valid JSON format. \nSummarization Style: Balanced\\nCreate a balanced summary that provides a moderate level of detail. Include the main points and key supporting information, while maintaining the document's overall structure. Aim for a comprehensive yet concise summary.\n Your output MUST be in valid JSON format with markdown content. You MUST strictly adhere to the output format specified in the instructions.
 assessment:
+  enabled: true
   image:
     target_height: ''
     target_width: ''