Merge branch 'develop' v0.3.14

Author: Bob Strahan

CHANGELOG.md

@@ -5,6 +5,27 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+## [0.3.14]
+
+### Added
+- Support for 1m token context for Claude Sonnet 4
+- Video demo of "Chat with Document" in [](./docs/web-ui.md)
+- **Human-in-the-Loop (HITL) Support Extended to Pattern-2**
+  - Added HITL review capabilities for Pattern-2 (Textract + Bedrock processing) using Amazon SageMaker Augmented AI (A2I)
+  - Enables human validation and correction when extraction confidence falls below configurable threshold
+  - Includes same features as Pattern-1 HITL: automatic triggering, review portal integration, and seamless result updates
+
+### Removed
+- Windows development environment guide and setup script removed as it proved insufficiently robust
+
+### Fixed
+- Fix 1-click Launch URL output from the GovCloud template generation script
+- Add Agent Analytics to architecture diagram
+- Fix various UX and error reporting issues with the new Python publish script
+- Simplify UDOP model path construction and avoid invalid default for regions other than us-east-1 and us-west-2
+- Permission regression from previous release affecting "Chat with Document"
+
+
 ## [0.3.13]
 
 ### Added

README.md

@@ -39,6 +39,8 @@ White-glove customization, deployment, and integration support for production us
 - **Cost Optimization**: Pay-per-use pricing model with built-in controls
 - **Comprehensive Monitoring**: Rich CloudWatch dashboard with detailed metrics and logs
 - **Web User Interface**: Modern UI for inspecting document workflow status and results
+- **Human-in-the-Loop (HITL)**: Amazon A2I integration for human review workflows (Pattern 1 & Pattern 2)
+  - **Note**: When deploying multiple patterns with HITL, reuse existing private workteam ARN due to AWS account limits
 - **AI-Powered Evaluation**: Framework to assess accuracy against baseline data
 - **Extraction Confidence Assessment**: LLM-powered assessment of extraction confidence with multimodal document analysis
 - **Document Knowledge Base Query**: Ask questions about your processed documents
@@ -128,6 +130,7 @@ For detailed deployment and testing instructions, see the [Deployment Guide](./d
 - [Configuration](./docs/configuration.md) - Configuration and customization options
 - [Classification](./docs/classification.md) - Customizing document classification
 - [Extraction](./docs/extraction.md) - Customizing information extraction
+- [Human-in-the-Loop Review](./docs/human-review.md) - Human review workflows with Amazon A2I
 - [Assessment](./docs/assessment.md) - Extraction confidence evaluation using LLMs
 - [Evaluation Framework](./docs/evaluation.md) - Accuracy assessment system with analytics database and reporting
 - [Knowledge Base](./docs/knowledge-base.md) - Document knowledge base query feature

VERSION

@@ -1 +1 @@
-0.3.14-wip
+0.3.14

docs/agent-analysis.md

@@ -37,15 +37,18 @@ https://github.com/user-attachments/assets/e2dea2c5-5eb1-42f6-9af5-469afd2135a7
 
 ## Architecture
 
+The architecture of the Agent Analysis feature is shown below. The Web UI and AppSync API components are the same as used by the rest of the IDP system (with new AppSync endpoints added). Note the inclusion of Amazon Athena and the AgentCore Code Interpreter is specific to the Analytics Agent, but the Agent Request Handler and Processor lambdas along with jobs table are used for all agents, including [MCP agents](./custom-MCP-agent.md).
+
+![Architecture Diagram](../images/IDP-AnalyticsAgent.drawio.png)
+
 ### Multi-Agent System
 
 The Agent Analysis feature uses a multi-agent architecture with:
 
 1. **Orchestrator Agent**: Routes queries to appropriate specialized agents based on query content and agent capabilities
 2. **Analytics Agent**: Handles data analysis, SQL generation, and visualization creation
-3. **Dummy Agent**: Provides simple calculations and development testing capabilities
-4. **External MCP Agents**: Custom agents connected via Model Context Protocol servers
-5. **Extensible Framework**: Easy addition of new specialized agents for specific use cases
+3. **External MCP Agents**: Custom agents connected via Model Context Protocol servers
+
 
 ### Agent Workflow
 

docs/custom-MCP-agent.md

@@ -20,23 +20,9 @@ The External MCP Agent allows you to:
 - **Real-time Integration**: Tools are available immediately through the IDP web interface
 
 ## Architecture
+An example architecture demonstrating the authentication flow and connections between the MCP Client (running in the IDP application) and an external MCP Server (deployed outside of the IDP application) can be seen below. The `get_client_address` and `send_verification_email` APIs drawn are just for demonstration purposes.
 
-```
-┌─────────────────────┐    ┌──────────────────────┐    ┌─────────────────────┐
-│   IDP Solution      │    │   Your AWS Account   │    │   Your MCP Server   │
-│                     │    │                      │    │                     │
-│ ┌─────────────────┐ │    │ ┌──────────────────┐ │    │ ┌─────────────────┐ │
-│ │ External MCP    │ │    │ │ AWS Cognito      │ │    │ │ Custom Tools    │ │
-│ │ Agent           │◄┼────┼─┤ User Pool        │◄┼────┼─┤ - Calculator    │ │
-│ │                 │ │    │ │                  │ │    │ │ - Weather API   │ │
-│ └─────────────────┘ │    │ └──────────────────┘ │    │ │ - Database      │ │
-│                     │    │                      │    │ │ - Custom Logic  │ │
-│ ┌─────────────────┐ │    │                      │    │ └─────────────────┘ │
-│ │ AWS Secrets     │ │    │                      │    │                     │
-│ │ Manager         │ │    │                      │    │                     │
-│ └─────────────────┘ │    │                      │    │                     │
-└─────────────────────┘    └──────────────────────┘    └─────────────────────┘
-```
+![Architecture Diagram](../images/IDP-external-mcp-example.drawio.png)
 
 ## Prerequisites
 

docs/govcloud-deployment.md

@@ -42,7 +42,7 @@ You need to have the following packages installed on your computer:
 3. [sam (AWS SAM)](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html)
 4. python 3.11 or later
 5. A local Docker daemon
-6. Python packages for publish.py: `pip install boto3 typer rich botocore setuptools`
+6. Python packages for publish.py: `pip install boto3 rich PyYAML botocore setuptools`
 
 ### Step 1: Generate GovCloud Template
 
@@ -65,7 +65,7 @@ aws cloudformation deploy \
   --template-file .aws-sam/idp-govcloud.yaml \
   --stack-name my-idp-govcloud-stack \
   --region us-gov-west-1 \
-  --capabilities CAPABILITY_IAM \
+  --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
   --parameter-overrides \
     IDPPattern="Pattern2 - Packet processing with Textract and Bedrock"
 ```

docs/human-review.md

@@ -0,0 +1,177 @@
+# Human-in-the-Loop (HITL) Review
+
+Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+SPDX-License-Identifier: MIT-0
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Architecture](#architecture)
+- [Workflow](#workflow)
+- [Configuration](#configuration)
+- [Review Portal](#review-portal)
+- [Best Practices](#best-practices)
+- [Troubleshooting](#troubleshooting)
+- [Known Limitations](#known-limitations)
+
+## Overview
+
+The GenAI-IDP solution supports Human-in-the-Loop (HITL) review capabilities using Amazon SageMaker Augmented AI (A2I). This feature enables human reviewers to validate and correct extracted information when the system's confidence falls below a specified threshold, ensuring accuracy for critical document processing workflows.
+
+**Supported Patterns:**
+- Pattern 1: BDA processing with HITL review
+- Pattern 2: Textract + Bedrock processing with HITL review
+
+
+https://github.com/user-attachments/assets/126c9a70-6811-46f3-9166-ef71397ea4bc
+
+
+
+## Architecture
+
+The HITL system integrates with the document processing workflow through:
+
+- **Amazon SageMaker A2I**: Manages human review tasks and workflows
+- **Private Workforce**: Secure reviewer access with existing credentials
+- **Review Portal**: Web interface for validation and correction
+- **Confidence Assessment**: Automated triggering based on extraction confidence scores
+
+<img src="../images/hitl_a2i_workflow.png" alt="HITL Flow diagram" width="800">
+
+## Workflow
+
+### 1. Automatic Triggering
+
+HITL review is automatically triggered when:
+- HITL feature is enabled in your configuration
+- Extraction confidence score falls below the configured threshold
+- The system creates a human review task in SageMaker A2I
+
+### 2. Review Process
+
+**Access:**
+- Reviewers access the SageMaker A2I Review Portal (URL available in CloudFormation output `SageMakerA2IReviewPortalURL`)
+- Login credentials are the same as those used for the GenAI IDP portal
+
+**Review Tasks:**
+- Extracted key-value pairs are presented for validation and correction
+- Reviewers can validate correct extractions or make necessary corrections
+- Submit corrections using the "Submit" button
+
+### 3. Result Integration
+
+- Corrected key-value pairs automatically update the source results
+- The document processing workflow continues with the human-verified data
+- Processing status is updated to reflect human review completion
+
+## Configuration
+
+### Deployment Parameters
+
+**Pattern 1:**
+- `EnableHITL`: Boolean parameter to enable/disable the HITL feature
+- `Pattern1 - Existing Private Workforce ARN`: parameter to use existing private workforce (reuse existing private workteam ARN due to AWS account limits)
+
+**Pattern 2:**
+- `EnableHITL`: Boolean parameter to enable/disable the HITL feature  
+- `Pattern2 - Existing Private Workforce ARN`: parameter to use existing private workforce (reuse existing private workteam ARN due to AWS account limits)
+
+### Confidence Threshold Configuration
+
+The confidence threshold determines when human review is triggered:
+
+1. **Access the Web UI**: Open the Web UI URL from your CloudFormation stack outputs
+2. **Navigate to Configuration**: Click on the "Configuration" tab in the navigation menu
+3. **Find Assessment & HITL Section**: Scroll to the "Assessment & HITL Configuration" section
+4. **Set Confidence Threshold**: 
+   - Enter a value between 0.0-1.0 (e.g., 0.8 for 80% confidence threshold)
+   - Fields with confidence scores below this threshold will trigger HITL review
+5. **Save Configuration**: Click "Save" to apply the changes
+
+The confidence threshold is stored as a configuration parameter and automatically applied to new document processing without requiring stack redeployment.
+
+## Review Portal
+
+### Accessing the Portal
+
+The SageMaker A2I Review Portal URL is available in your CloudFormation stack outputs as `SageMakerA2IReviewPortalURL`.
+
+### Portal Features
+
+- **Task Queue**: View all pending review tasks
+- **Document Preview**: Visual representation of the document being reviewed
+- **Key-Value Editor**: Interface for validating and correcting extracted data
+- **Submission Controls**: Submit approved or corrected extractions
+
+### Reviewer Credentials
+
+- Use the same credentials as the GenAI IDP Web UI portal
+- If using an existing private workforce, provide the workforce ARN during deployment
+
+## Best Practices
+
+### Review Management
+- **Regular Monitoring**: Check the Review Portal regularly for pending tasks to avoid processing delays
+- **Consistent Guidelines**: Establish consistent correction guidelines if multiple reviewers are involved
+- **Quality Assurance**: Implement review quality checks for critical document types
+
+### Threshold Optimization
+- **Start Conservative**: Begin with higher confidence thresholds (0.8-0.9) and adjust based on accuracy needs
+- **Monitor Performance**: Track review frequency and accuracy improvements to optimize thresholds
+- **Document-Specific**: Consider different thresholds for different document types based on complexity
+
+### Workflow Integration
+- **Training**: Ensure reviewers understand the document types and expected extraction fields
+- **Escalation**: Define processes for complex cases that require additional expertise
+- **Feedback Loop**: Use review corrections to improve extraction prompts and configurations
+
+## Troubleshooting
+
+### Common Issues
+
+**No Review Tasks Appearing:**
+- Verify HITL is enabled in deployment parameters
+- Check confidence threshold settings in Web UI configuration
+- Ensure documents are triggering confidence scores below threshold
+
+**Portal Access Issues:**
+- Verify reviewer credentials match GenAI IDP Web UI credentials
+- Check private workforce configuration if using existing workforce
+- Confirm portal URL from CloudFormation outputs
+
+**Review Submissions Not Processing:**
+- Check Step Functions execution for error details
+- Verify A2I workflow definition is properly configured
+- Review CloudWatch logs for processing errors
+
+### Monitoring
+
+Monitor HITL performance through:
+- **CloudWatch Metrics**: Track review task creation and completion rates
+- **Step Functions**: Monitor workflow execution and HITL integration points
+- **Web UI Dashboard**: View document processing status including HITL stages
+
+## Known Limitations
+
+### Current Limitations
+
+- **Task Navigation**: Current version of SageMaker A2I cannot provide direct hyperlink to specific document tasks. When reviewers click on review document URL, the portal displays all review tasks without task-specific navigation.
+
+- **Template Updates**: Updating SageMaker A2I Template and workflow performs deletion on A2I flow definition and custom template, then recreates resources via Lambda function. Direct updates to A2I resources through Python SDK are not supported.
+
+- **Private Workforce Cognito Limitation**: AWS SageMaker Ground Truth allows only **one private workforce per Cognito User Pool** per AWS account. This creates a critical dependency when deploying multiple GenAI-IDP stacks with HITL enabled:
+  - Each private workforce must be mapped to a unique Cognito client
+  - Multiple stacks cannot create separate private workforces if they use the same Cognito User Pool
+  - **Risk**: If the first stack (that created the private workforce) is deleted, it will break the private workteam for all other stacks using the same workforce
+  - **Recommendation**: Always reuse existing private workteam ARNs when deploying additional patterns or stacks with HITL enabled
+  - Use the `ExistingPrivateWorkforceArn` parameter to reference the workforce created by your first HITL-enabled deployment
+
+### Workarounds
+
+- **Task Management**: Reviewers should process tasks in chronological order or use task identifiers to track specific documents
+- **Configuration Changes**: Plan A2I template updates during maintenance windows to minimize disruption
+- **Multi-Stack HITL Deployment**: 
+  1. Deploy your first HITL-enabled stack and note the `PrivateWorkteamArn` from CloudFormation outputs
+  2. For subsequent stacks, provide this ARN in the `ExistingPrivateWorkforceArn` parameter
+  3. Never delete the original stack that created the private workforce without first migrating the workforce to another stack
+  4. Consider creating a dedicated "HITL infrastructure" stack to manage the private workforce independently

docs/pattern-1.md

@@ -196,49 +196,11 @@ payload = {
 
 Pattern-1 supports Human-in-the-Loop (HITL) review capabilities using Amazon SageMaker Augmented AI (A2I). This feature allows human reviewers to validate and correct extracted information when the system's confidence falls below a specified threshold.
 
-#### HITL Workflow
-1. **Automatic Triggering**: 
-   - HITL is triggered when the feature is enabled in your configuration
-   - Extraction confidence score falls below your configured confidence threshold
-   - The system creates a human review task in SageMaker A2I
-
-2. **Review Process**:
-   - Reviewers access the SageMaker A2I Review Portal (URL available in CloudFormation output `SageMakerA2IReviewPortalURL`)
-   - Login credentials are the same as those used for the GenAI IDP portal (if you want to use your own Private work team, you can provide your existing private workforce work team arn as a input parameter for `Pattern1 - Existing Private Workforce ARN`)
-   - Extracted key-value pairs are presented for validation and correction
-   - Reviewers validate correct extractions or make necessary corrections
-   - After review, corrections are submitted with the "Submit" button
-
-3. **Result Integration**:
-   - Corrected key-value pairs automatically update the source results
-   - The document processing workflow continues with the human-verified data
-
-<img src="../../images/hitl_a2i_workflow.png" alt="HITL Flow diagram" width="800">
-
-#### Configuration
+**Pattern-1 Specific Configuration:**
 - `EnableHITL`: Boolean parameter to enable/disable the HITL feature
-- **Confidence Threshold**: Configured through the Web UI Portal Configuration tab under "Assessment & HITL Configuration" section. This numeric value (0.0-1.0) determines when human review is triggered based on extraction confidence scores.
+- `Pattern1 - Existing Private Workforce ARN`: Optional parameter to use existing private workforce
 
-#### Configuring Confidence Threshold
-To set the confidence threshold for HITL triggering:
-
-1. **Access the Web UI**: Open the Web UI URL from your CloudFormation stack outputs
-2. **Navigate to Configuration**: Click on the "Configuration" tab in the navigation menu
-3. **Find Assessment & HITL Section**: Scroll to the "Assessment & HITL Configuration" section
-4. **Set Confidence Threshold**: 
-   - Enter a value between 0.0-1.0 (e.g., 0.8 for 80% confidence threshold)
-   - Fields with confidence scores below this threshold will trigger HITL review
-5. **Save Configuration**: Click "Save" to apply the changes
-
-The confidence threshold is stored as a configuration parameter and automatically applied to new document processing without requiring stack redeployment.
-
-#### Best Practices
-- Regularly check the Review Portal for pending tasks to avoid processing delays
-- Establish consistent correction guidelines if multiple reviewers are involved
-
-#### Known Limitations
-  - Current version of SageMaker A2I cannot provide direct hyperlink to respective document tasks. When reviewer clicks on review document URL and start working, review portal will start displating all review tasks. Reviewer cannot pick specific task and start working. 
-  - Updating SageMaker A2I Template and workflow performs deletion on A2I flow definition, A2I custom template and recreate the resources via lambda function. Update A2I resources through Python SDK is not allowed. 
+For comprehensive HITL documentation including workflow details, configuration steps, best practices, and troubleshooting, see the [Human-in-the-Loop Review Guide](./human-review.md). 
 
 ## Best Practices
 1. **BDA Project Configuration**: