Feature/stickler

Author: rstrahan

CHANGELOG.md

@@ -5,17 +5,36 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+### Added
+
 ## [0.4.2]
 
 ### Added
 
+- **Stickler-Based Evaluation System for Enhanced Comparison Capabilities**
+  - Migrated evaluation service from custom comparison logic to [AWS Labs Stickler library](https://github.com/awslabs/stickler/tree/main) for structured object evaluation
+  - **Field Importance Weights**: New capability to assign business criticality weights to fields (e.g., shipment ID weight=3.0 vs notes weight=0.5)
+  - **Enhanced Configuration**: Added `x-aws-idp-evaluation-*` extensions for evaluation configuration
+  - **Backward compatible**: Maintained API compatibility - all existing code works unchanged
+  - **Enhanced Comparators**: Leverages Stickler's optimized comparison algorithms (Exact, Levenshtein, Numeric, Fuzzy, Semantic) with LLM evaluation preserved through custom wrapper
+  - **Better List Matching**: Hungarian algorithm via Stickler for optimal list comparisons regardless of order
+
+- **UI: Evaluation Configuration in Document Schema UI**
+  - Added evaluation weight, threshold (with conditional display), and document-level match threshold fields for complete Stickler configuration control
+  - Added LEVENSHTEIN and HUNGARIAN evaluation methods with auto-populated threshold defaults based on selected method
+  
 - **IDP CLI Force Delete All Resources Option**
   - Added `--force-delete-all` flag to `idp-cli delete` command for comprehensive stack cleanup
   - **Post-CloudFormation Cleanup**: Analyzes resources after CloudFormation deletion completes to identify retained resources (DELETE_SKIPPED status)
   - **Use Cases**: Complete test environment cleanup, CI/CD pipelines requiring full teardown, cost optimization by removing all retained resources
 
 ### Changed
 
+- **Containerized Pattern-1 and Pattern-3 Deployment Pipelines**
+  - Migrated Pattern-1 and Pattern-3 Lambda functions to Docker image deployments (following Pattern-2 approach from v0.3.20)
+  - Builds and pushes all Lambda images via CodeBuild with automated ECR cleanup
+  - Increases Lambda package size limit from 250 MB (zip) to 10 GB (Docker image) to accommodate larger dependencies
+
 - **Agent Companion Chat - Chat History Feature**
   - Added chat history feature from Agent Analysis back into Agent Companion Chat
   - Users can now load and view previous chat sessions with full conversation context
@@ -28,9 +47,18 @@ SPDX-License-Identifier: MIT-0
   - Prompt input is disabled during active streaming responses to prevent concurrent requests
   - Fixed issue where charts in loaded chat history were not displaying
 
-- **GovCloud Template Generation - Missing Chat Resources**
+- **GovCloud Template Generation errors**
   - Fixed CloudFormation deployment error `Fn::GetAtt references undefined resource GraphQLApi` when deploying GovCloud templates
 
+- **Example Notebook error fixed**
+  - Example notebooks updated to work with new v0.4.0+ JSON schema
+
+
+### Templates
+   - us-west-2: `https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.4.2yaml`
+   - us-east-1: `https://s3.us-east-1.amazonaws.com/aws-ml-blog-us-east-1/artifacts/genai-idp/idp-main_0.4.2.yaml`
+   - eu-central-1: `https://s3.eu-central-1.amazonaws.com/aws-ml-blog-eu-central-1/artifacts/genai-idp/idp-main_0.4.2.yaml`
+
 ## [0.4.1]
 
 ### Changed

Makefile

@@ -16,6 +16,7 @@ test:
 
 # Run both linting and formatting in one command
 lint: ruff-lint format check-arn-partitions validate-buildspec ui-lint
+fastlint: ruff-lint format check-arn-partitions validate-buildspec
 
 # Run linting checks and fix issues automatically
 ruff-lint:
@@ -123,3 +124,10 @@ commit: lint test
 	git add . && \
 	git commit -am "$${COMMIT_MESSAGE}" && \
 	git push
+
+fastcommit: fastlint
+	$(info Generating commit message...)
+	export COMMIT_MESSAGE="$(shell q chat --no-interactive --trust-all-tools "Understand pending local git change and changes to be committed, then infer a commit message. Return this commit message only" | tail -n 1 | sed 's/\x1b\[[0-9;]*m//g')" && \
+	git add . && \
+	git commit -am "$${COMMIT_MESSAGE}" && \
+	git push

VERSION

@@ -1 +1 @@
-0.4.2-wip2
+0.4.2-rc1

docs/README.md

@@ -13,7 +13,7 @@ This folder contains detailed documentation on various aspects of the GenAI Inte
 - [Agent Analysis](./agent-analysis.md) - Natural language analytics and data visualization feature
 - [Knowledge Base](./knowledge-base.md) - Document knowledge base query feature
 - [Post-Processing Lambda Hook](./post-processing-lambda-hook.md) - Custom downstream processing integration
-- [Evaluation Framework](./evaluation.md) - Accuracy assessment system
+- [Evaluation Framework](./evaluation.md) - Accuracy assessment system powered by Stickler
 - [Assessment Feature](./assessment.md) - Extraction confidence evaluation using LLMs
 - [Configuration](./configuration.md) - Configuration and customization options
 - [Classification](./classification.md) - Customizing document classification

docs/configuration.md

@@ -312,6 +312,63 @@ Status lookup returns comprehensive information:
 }
 ```
 
+## Evaluation Extensions in JSON Schema
+
+Document class schemas support evaluation-specific extensions for fine-grained control over accuracy assessment. These extensions work with the [Stickler](https://github.com/awslabs/stickler)-based evaluation framework to provide flexible, business-aligned evaluation capabilities.
+
+### Available Extensions
+
+- `x-aws-idp-evaluation-method`: Comparison method (EXACT, FUZZY, NUMERIC_EXACT, SEMANTIC, LLM, HUNGARIAN)
+- `x-aws-idp-evaluation-threshold`: Minimum score to consider a match (0.0-1.0)
+- `x-aws-idp-evaluation-weight`: Field importance for weighted scoring (default: 1.0, higher values = more important)
+
+### Example Configuration
+
+```yaml
+classes:
+  - $schema: "https://json-schema.org/draft/2020-12/schema"
+    x-aws-idp-document-type: "Invoice"
+    x-aws-idp-evaluation-match-threshold: 0.8  # Document-level threshold
+    properties:
+      invoice_number:
+        type: string
+        x-aws-idp-evaluation-method: EXACT
+        x-aws-idp-evaluation-weight: 2.0  # Critical field - double weight
+      invoice_date:
+        type: string
+        x-aws-idp-evaluation-method: FUZZY
+        x-aws-idp-evaluation-threshold: 0.9
+        x-aws-idp-evaluation-weight: 1.5  # Important field
+      vendor_name:
+        type: string
+        x-aws-idp-evaluation-method: FUZZY
+        x-aws-idp-evaluation-threshold: 0.85
+        x-aws-idp-evaluation-weight: 1.0  # Normal weight (default)
+      vendor_notes:
+        type: string
+        x-aws-idp-evaluation-method: SEMANTIC
+        x-aws-idp-evaluation-threshold: 0.7
+        x-aws-idp-evaluation-weight: 0.5  # Less critical - half weight
+```
+
+### Stickler Backend Integration
+
+The evaluation framework uses [Stickler](https://github.com/awslabs/stickler) as its evaluation engine. The `SticklerConfigMapper` automatically translates these IDP extensions to Stickler's native format, providing:
+
+- **Field-level weighting** for business-critical attributes
+- **Optimal list matching** using the Hungarian algorithm
+- **Extensible comparator system** with exact, fuzzy, numeric, semantic, and LLM-based comparison
+- **Native JSON Schema support** with $ref resolution
+
+### Benefits
+
+1. **Business Alignment**: Weight critical fields higher to ensure evaluation scores reflect business priorities
+2. **Flexible Comparison**: Choose the right evaluation method for each field type
+3. **Tunable Thresholds**: Set field-specific thresholds for matching sensitivity
+4. **Dynamic Schema Generation**: Auto-generates evaluation schema from baseline data when configuration is missing (for development/prototyping)
+
+For detailed evaluation capabilities and best practices, see [evaluation.md](evaluation.md).
+
 ## Cost Tracking and Optimization
 
 The solution includes built-in cost tracking capabilities: