Merge remote-tracking branch 'origin/develop' into feature/mcp-server

Author: webarch-ai

CHANGELOG.md

@@ -5,7 +5,43 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+### Added
+
+- **Amazon Nova 2 Lite Model Support**
+  - Added support for Amazon Nova 2 Lite model (`us.amazon.nova-2-lite-v1:0`, `eu.amazon.nova-2-lite-v1:0`)
+  - Available for configuration across all document processing steps
+  - Added to prompt caching supported models list
+
+- **Anthropic Claude Opus 4.5 Model Support**
+  - Added support for Claude Opus 4.5 model (`us.anthropic.claude-opus-4-5-20251101-v1:0`, `eu.anthropic.claude-opus-4-5-20251101-v1:0`)
+  - Available for configuration across all document processing steps
+  - Added to prompt caching supported models list
+
+- **Qwen Model Support**
+  - Added support for Qwen 3 VL model (`qwen.qwen3-vl-235b-a22b`)
+  - Available for configuration in document processing workflows
+
+- **Configurable Section Splitting Strategies for Enhanced Document Segmentation Control**
+  - Added new `sectionSplitting` configuration option to control how classified pages are grouped into document sections
+  - **Three Strategies Available**:
+    - `disabled`: Entire document treated as single section with first detected class (simplest case)
+    - `page`: One section per page preventing automatic joining of same-type documents (deterministic, solves Issue #146)
+    - `llm_determined`: Uses LLM boundary detection with "Start"/"Continue" indicators (default, maintains existing behavior)
+  - **Key Benefits**: Deterministic splitting for long documents with multiple same-type forms (e.g., multiple W-2s, multiple invoices), eliminates LLM boundary detection failures for critical government form processing, provides flexibility across simple to complex document scenarios
+  - Resolves #146
+
 ### Changed
+
+- **Improved Temperature and Top_P Parameter Logic for Deterministic Output**
+  - Changed inference parameter selection logic to allow `temperature=0.0` for deterministic output (recommended by Anthropic and other model providers)
+  - **New Logic**: Uses `top_p` only when it has a positive value (> 0); otherwise uses `temperature` including `temperature=0.0`
+  - **Previous Logic**: Used `top_p` whenever `temperature=0.0`, preventing proper deterministic configuration
+  - **Key Benefits**: Enables proper deterministic output with `temperature=0.0`, more intuitive parameter behavior, aligns with model provider best practices (Anthropic recommends `temperature=0` for consistent outputs)
+  - **Affected Components**: Bedrock client (`lib/idp_common_pkg/idp_common/bedrock/client.py`), Agentic extraction service (`lib/idp_common_pkg/idp_common/extraction/agentic_idp.py`)
+  - **Configuration Guidance**: Set `top_p: 0` to use `temperature` parameter; set `top_p` to positive value to override temperature
+  - Set temperature to 0.0 in discovery config for deterministic discovery output (was previously set to 1.0)
+  - Set top_p to 0.0 in all repo config files to force use of temperature setting by default.
+
 - Removed page image limit entirely across all IDP services (classification, extraction, assessment) following Amazon Bedrock API removal of image count restrictions. The system now processes all document pages without artificial truncation, with info logging to track image counts for monitoring purposes.
   - Resolves #147
 

config_library/pattern-1/lending-package-sample/config.yaml

@@ -6,7 +6,7 @@ assessment:
   default_confidence_threshold: '0.8'
 summarization:
   enabled: true
-  top_p: '0.1'
+  top_p: "0.0"
   max_tokens: '4096'
   top_k: '5'
   task_prompt: >-
@@ -62,14 +62,14 @@ summarization:
 
     Do not include any text, explanations, or notes outside of this JSON
     structure. The JSON must be properly formatted and parseable.
-  temperature: '0.0'
+  temperature: "0.0"
   model: us.amazon.nova-premier-v1:0
   system_prompt: >-
     You are a document summarization expert who can analyze and summarize documents from various domains including medical, financial, legal, and general business documents. Your task is to create a summary that captures the key information, main points, and important details from the document. Your output must be in valid JSON format. \nSummarization Style: Balanced\\nCreate a balanced summary that provides a moderate level of detail. Include the main points and key supporting information, while maintaining the document's overall structure. Aim for a comprehensive yet concise summary.\n Your output MUST be in valid JSON format with markdown content. You MUST strictly adhere to the output format specified in the instructions.
 evaluation:
   enabled: true
   llm_method:
-    top_p: '0.1'
+    top_p: "0.0"
     max_tokens: '4096'
     top_k: '5'
     task_prompt: >-
@@ -101,7 +101,7 @@ evaluation:
         "score": 0.0 to 1.0,
         "reason": "Your explanation here"
       }
-    temperature: '0.0'
+    temperature: "0.0"
     model: us.anthropic.claude-3-haiku-20240307-v1:0
     system_prompt: >-
       You are an evaluator that helps determine if the predicted and expected values match for document attribute extraction. You will consider the context and meaning rather than just exact string matching.
@@ -151,8 +151,8 @@ discovery:
           ]
       }
   with_ground_truth:
-    top_p: '0.1'
-    temperature: '1.0'
+    top_p: "0.0"
+    temperature: "0.0"
     user_prompt: >-
       This image contains unstructured data. Analyze the data line by line using the provided ground truth as reference.                        
       <GROUND_TRUTH_REFERENCE>
@@ -179,8 +179,8 @@ discovery:
       field definitions.
     max_tokens: '10000'
   without_ground_truth:
-    top_p: '0.1'
-    temperature: '1.0'
+    top_p: "0.0"
+    temperature: "0.0"
     user_prompt: >-
       This image contains forms data. Analyze the form line by line.
       Image may contains multiple pages, process all the pages. 
@@ -336,6 +336,16 @@ pricing:
         price: '2.5E-6'
       - name: outputTokens
         price: '1.25E-5'
+  - name: bedrock/us.amazon.nova-2-lite-v1:0
+    units:
+      - name: inputTokens
+        price: '3.0E-7'
+      - name: outputTokens
+        price: '2.5E-6'
+      - name: cacheReadInputTokens
+        price: '7.5E-8'
+      - name: cacheWriteInputTokens
+        price: '3.0E-7'
   - name: bedrock/us.anthropic.claude-3-haiku-20240307-v1:0
     units:
       - name: inputTokens
@@ -442,6 +452,16 @@ pricing:
         price: '1.5E-6'
       - name: cacheWriteInputTokens
         price: '1.875E-5'
+  - name: bedrock/us.anthropic.claude-opus-4-5-20251101-v1:0
+    units:
+      - name: inputTokens
+        price: '5.0E-06'
+      - name: outputTokens
+        price: '2.5E-05'
+      - name: cacheReadInputTokens
+        price: '5.0E-07'
+      - name: cacheWriteInputTokens
+        price: '6.25E-06'
   # EU model pricing
   - name: bedrock/eu.amazon.nova-lite-v1:0
     units:
@@ -463,6 +483,16 @@ pricing:
         price: '2.6E-7'
       - name: cacheWriteInputTokens
         price: '1.0E-6'
+  - name: bedrock/eu.amazon.nova-2-lite-v1:0
+    units:
+      - name: inputTokens
+        price: '3.9E-7'
+      - name: outputTokens
+        price: '3.27E-6'
+      - name: cacheReadInputTokens
+        price: '9.75E-8'
+      - name: cacheWriteInputTokens
+        price: '3.9E-7'
   - name: bedrock/eu.anthropic.claude-3-haiku-20240307-v1:0
     units:
       - name: inputTokens
@@ -529,6 +559,22 @@ pricing:
         price: '6.6E-7'
       - name: cacheWriteInputTokens
         price: '8.25E-6'
+  - name: bedrock/eu.anthropic.claude-opus-4-5-20251101-v1:0
+    units:
+      - name: inputTokens
+        price: '5.0E-6'
+      - name: outputTokens
+        price: '2.5E-5'
+      - name: cacheReadInputTokens
+        price: '5.0E-7'
+      - name: cacheWriteInputTokens
+        price: '6.25E-6'
+  - name: bedrock/qwen.qwen3-vl-235b-a22b
+    units:
+      - name: inputTokens
+        price: '5.3E-7'
+      - name: outputTokens
+        price: '2.66E-6'
   # AWS Lambda pricing (US East - N. Virginia)
   - name: lambda/requests
     units:

config_library/pattern-2/bank-statement-sample/config.yaml

@@ -100,7 +100,7 @@ classification:
   image:
     target_height: ""
     target_width: ""
-  top_p: "0.1"
+  top_p: "0.0"
   max_tokens: "4096"
   top_k: "5"
   task_prompt: >-
@@ -238,11 +238,12 @@ classification:
   system_prompt: >-
     You are a document classification expert who can analyze and classify multiple documents and their page boundaries within a document package from various domains. Your task is to determine the document type based on its content and structure, using the provided document type definitions. Your output must be valid JSON according to the requested format.
   classificationMethod: textbasedHolisticClassification
+  sectionSplitting: llm_determined
 extraction:
   image:
     target_height: ""
     target_width: ""
-  top_p: "0.1"
+  top_p: "0.0"
   max_tokens: "10000"
   top_k: "5"
   task_prompt: >-
@@ -337,7 +338,7 @@ extraction:
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
   enabled: true
-  top_p: "0.1"
+  top_p: "0.0"
   max_tokens: "4096"
   top_k: "5"
   task_prompt: >-
@@ -410,7 +411,7 @@ assessment:
     simple_batch_size: "3"
     list_batch_size: "1"
   default_confidence_threshold: "0.8"
-  top_p: "0.1"
+  top_p: "0.0"
   max_tokens: "10000"
   top_k: "5"
   temperature: "0.0"
@@ -559,7 +560,7 @@ assessment:
 evaluation:
   enabled: true
   llm_method:
-    top_p: "0.1"
+    top_p: "0.0"
     max_tokens: "4096"
     top_k: "5"
     task_prompt: >-
@@ -641,8 +642,8 @@ discovery:
           ]
       }
   with_ground_truth:
-    top_p: "0.1"
-    temperature: "1.0"
+    top_p: "0.0"
+    temperature: "0.0"
     user_prompt: >-
       This image contains unstructured data. Analyze the data line by line using the provided ground truth as reference.                        
       <GROUND_TRUTH_REFERENCE>
@@ -669,8 +670,8 @@ discovery:
       field definitions.
     max_tokens: "10000"
   without_ground_truth:
-    top_p: "0.1"
-    temperature: "1.0"
+    top_p: "0.0"
+    temperature: "0.0"
     user_prompt: >-
       This image contains forms data. Analyze the form line by line.
       Image may contains multiple pages, process all the pages. 
@@ -842,6 +843,16 @@ pricing:
         price: "2.5E-6"
       - name: outputTokens
         price: "1.25E-5"
+  - name: bedrock/us.amazon.nova-2-lite-v1:0
+    units:
+      - name: inputTokens
+        price: '3.0E-7'
+      - name: outputTokens
+        price: '2.5E-6'
+      - name: cacheReadInputTokens
+        price: '7.5E-8'
+      - name: cacheWriteInputTokens
+        price: '3.0E-7'
   - name: bedrock/us.anthropic.claude-3-haiku-20240307-v1:0
     units:
       - name: inputTokens
@@ -948,6 +959,16 @@ pricing:
         price: "1.5E-6"
       - name: cacheWriteInputTokens
         price: "1.875E-5"
+  - name: bedrock/us.anthropic.claude-opus-4-5-20251101-v1:0
+    units:
+      - name: inputTokens
+        price: '5.0E-06'
+      - name: outputTokens
+        price: '2.5E-05'
+      - name: cacheReadInputTokens
+        price: '5.0E-07'
+      - name: cacheWriteInputTokens
+        price: '6.25E-06'
   # EU model pricing
   - name: bedrock/eu.amazon.nova-lite-v1:0
     units:
@@ -969,6 +990,16 @@ pricing:
         price: "2.6E-7"
       - name: cacheWriteInputTokens
         price: "1.0E-6"
+  - name: bedrock/eu.amazon.nova-2-lite-v1:0
+    units:
+      - name: inputTokens
+        price: '3.9E-7'
+      - name: outputTokens
+        price: '3.27E-6'
+      - name: cacheReadInputTokens
+        price: '9.75E-8'
+      - name: cacheWriteInputTokens
+        price: '3.9E-7'
   - name: bedrock/eu.anthropic.claude-3-haiku-20240307-v1:0
     units:
       - name: inputTokens
@@ -1035,6 +1066,22 @@ pricing:
         price: "6.6E-7"
       - name: cacheWriteInputTokens
         price: "8.25E-6"
+  - name: bedrock/eu.anthropic.claude-opus-4-5-20251101-v1:0
+    units:
+      - name: inputTokens
+        price: '5.0E-6'
+      - name: outputTokens
+        price: '2.5E-5'
+      - name: cacheReadInputTokens
+        price: '5.0E-7'
+      - name: cacheWriteInputTokens
+        price: '6.25E-6'
+  - name: bedrock/qwen.qwen3-vl-235b-a22b
+    units:
+      - name: inputTokens
+        price: '5.3E-7'
+      - name: outputTokens
+        price: '2.66E-6'
   # AWS Lambda pricing (US East - N. Virginia)
   - name: lambda/requests
     units: