Merge branch 'develop' v0.3.5

Author: Bob Strahan

.gitignore

@@ -15,4 +15,5 @@ build/
 __pycache__
 *.code-workspace
 .ruff_cache
+.kiro
 rvl_cdip_*

.gitlab-ci.yml

@@ -28,6 +28,8 @@ developer_tests:
     - apt-get update -y
     - apt-get install make -y
     - pip install ruff
+    # Install test dependencies
+    - cd lib/idp_common_pkg && pip install -e ".[test]" && cd ../..
 
   script:
     - make lint-cicd

1751146101381_classification_state.json

@@ -5,6 +5,76 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+## [0.3.5]
+
+### Added
+- **Human-in-the-Loop (HITL) Support - Pattern 1**
+  - Added comprehensive Human-in-the-Loop review capabilities using Amazon SageMaker Augmented AI (A2I)
+  - **Key Features**:
+    - Automatic triggering when extraction confidence falls below configurable threshold
+    - Integration with SageMaker A2I Review Portal for human validation and correction
+    - Configurable confidence threshold through Web UI Portal Configuration tab (0.0-1.0 range)
+    - Seamless result integration with human-verified data automatically updating source results
+  - **Workflow Integration**: 
+    - HITL tasks created automatically when confidence thresholds are not met
+    - Reviewers can validate correct extractions or make necessary corrections through the Review Portal
+    - Document processing continues with human-verified data after review completion
+  - **Configuration Management**:
+    - `EnableHITL` parameter for feature toggle
+    - Confidence threshold configurable via Web UI without stack redeployment
+    - Support for existing private workforce work teams via input parameter
+  - **CloudFormation Output**: Added `SageMakerA2IReviewPortalURL` for easy access to review portal
+  - **Known Limitations**: Current A2I version cannot provide direct hyperlinks to specific document tasks; template updates require resource recreation
+- **Document Compression for Large Documents - all patterns**
+  - Added automatic compression support to handle large documents and avoid exceeding Step Functions payload limits (256KB)
+  - **Key Features**:
+    - Automatic compression (default trigger threshold of 0KB enables compression by default)
+    - Transparent handling of both compressed and uncompressed documents in Lambda functions
+    - Temporary S3 storage for compressed document state with automatic cleanup via lifecycle policies
+  - **New Utility Methods**:
+    - `Document.load_document()`: Automatically detects and decompresses document input from Lambda events
+    - `Document.serialize_document()`: Automatically compresses large documents for Lambda responses
+    - `Document.compress()` and `Document.decompress()`: Compression/decompression methods
+  - **Lambda Function Integration**: All relevant Lambda functions updated to use compression utilities
+  - **Resolves Step Functions Errors**: Eliminates "result with a size exceeding the maximum number of bytes service limit" errors for large multi-page documents
+- **Multi-Backend OCR Support - Pattern 2 and 3**
+  - Textract Backend (default): Existing AWS Textract functionality
+  - Bedrock Backend: New LLM-based OCR using Claude/Nova models
+  - None Backend: Image-only processing without OCR
+- **Bedrock OCR Integration - Pattern 2 and 3**
+  - Customizable system and task prompts for OCR optimization
+  - Better handling of complex documents, tables, and forms
+  - Layout preservation capabilities
+- **Image Preprocessing - Pattern 2 and 3**
+  - Adaptive Binarization: Improves OCR accuracy on documents with:
+    - Uneven lighting or shadows
+    - Low contrast text
+    - Background noise or gradients
+  - Optional feature with configurable enable/disable
+- **YAML Parsing Support for LLM Responses - Pattern 2 and 3**
+  - Added comprehensive YAML parsing capabilities to complement existing JSON parsing functionality
+  - New `extract_yaml_from_text()` function with robust multi-strategy YAML extraction:
+    - YAML in ```yaml and ```yml code blocks
+    - YAML with document markers (---)
+    - Pattern-based YAML detection using indentation and key indicators
+  - New `detect_format()` function for automatic format detection returning 'json', 'yaml', or 'unknown'
+  - New unified `extract_structured_data_from_text()` wrapper function that automatically detects and parses both JSON and YAML formats
+  - **Token Efficiency**: YAML typically uses 10-30% fewer tokens than equivalent JSON due to more compact syntax
+  - **Service Integration**: Updated classification service to use the new unified parsing function with automatic fallback between formats
+  - **Comprehensive Testing**: Added 39 new unit tests covering all YAML extraction strategies, format detection, and edge cases
+  - **Backward Compatibility**: All existing JSON functionality preserved unchanged, new functionality is purely additive
+  - **Intelligent Fallback**: Robust fallback mechanism handles cases where preferred format fails (e.g., JSON requested as YAML falls back to JSON)
+  - **Production Ready**: Handles malformed content gracefully, comprehensive error handling and logging
+  - **Example Notebook**: Added `notebooks/examples/step3_extraction_using_yaml.ipynb` demonstrating YAML-based extraction with automatic format detection and token efficiency benefits
+
+### Fixed
+- **Enhanced JSON Extraction from LLM Responses (Issue #16)**
+  - Modularized duplicate `_extract_json()` functions across classification, extraction, summarization, and assessment services into a common `extract_json_from_text()` utility function
+  - Improved multi-line JSON handling with literal newlines in string values that previously caused parsing failures
+  - Added robust JSON validation and multiple fallback strategies for better extraction reliability
+  - Enhanced string parsing with proper escape sequence handling for quotes and newlines
+  - Added comprehensive unit tests covering various JSON formats including multi-line scenarios
+
 ## [0.3.4]
 
 ### Added

CHANGELOG.md

@@ -46,4 +46,4 @@ commit: lint test
 	export COMMIT_MESSAGE="$(shell q chat --no-interactive --trust-all-tools "Understand pending local git change and changes to be committed, then infer a commit message. Return this commit message only" | tail -n 1 | sed 's/\x1b\[[0-9;]*m//g')" && \
 	git add . && \
 	git commit -am "$${COMMIT_MESSAGE}" && \
-	git push
+	git push

Makefile

@@ -74,7 +74,7 @@ After deployment, you can quickly process a document and view results:
 
 2. **Use Sample Documents**:
    - For Pattern 1 (BDA): Use [samples/lending_package.pdf](./samples/lending_package.pdf)
-   - For Patterns 2 and 3: Use [samples/rvl_cdip_package.pdf](./samples/rvl_cdip_package.pdf) 
+   - For Patterns 2 and 3: Use [samples/rvl_cdip_package.pdf](./samples/rvl_cdip_package.pdf)
 
 3. **Monitor Processing**:
    - **Via Web UI**: Track document status on the dashboard

README.md

@@ -1 +1 @@
-0.3.4
+0.3.5-delta

VERSION

@@ -2,6 +2,8 @@
 # SPDX-License-Identifier: MIT-0
 
 notes: Processing configuration in BDA project.
+assessment:
+  default_confidence_threshold: '0.8'
 summarization:
   top_p: '0.1'
   max_tokens: '4096'

config_library/pattern-1/default/config.yaml

@@ -3,8 +3,15 @@
 
 notes: Default settings
 ocr:
+  backend: "textract"  # Default to Textract for backward compatibility
+  model_id: "us.anthropic.claude-3-7-sonnet-20250219-v1:0"
+  system_prompt: "You are an expert OCR system. Extract all text from the provided image accurately, preserving layout where possible."
+  task_prompt: "Extract all text from this document image. Preserve the layout, including paragraphs, tables, and formatting."
   features:
     - name: LAYOUT
+  image:
+    target_width: '951'
+    target_height: '1268'
 classes:
   - name: Bank Statement
     description: Monthly bank account statement

config_library/pattern-2/bank-statement-sample/config.yaml

@@ -3,6 +3,10 @@
 
 notes: Default settings
 ocr:
+  backend: "textract"  # Default to Textract for backward compatibility
+  model_id: "us.anthropic.claude-3-7-sonnet-20250219-v1:0"
+  system_prompt: "You are an expert OCR system. Extract all text from the provided image accurately, preserving layout where possible."
+  task_prompt: "Extract all text from this document image. Preserve the layout, including paragraphs, tables, and formatting."
   features:
     - name: LAYOUT
     - name: TABLES