Merge branch 'develop' into feature/automate-dsr-checks

Author: Taniya Mathur

.gitignore

@@ -4,6 +4,8 @@ build.toml
 model.tar.gz
 .checksum
 .checksums/
+.build_checksum
+.lib_checksum
 .vscode/
 .DS_Store
 dist/

.gitlab-ci.yml

@@ -28,6 +28,8 @@ developer_tests:
     - apt-get update -y
     - apt-get install make -y
     - pip install ruff
+    # Install dependencies needed by publish.py for test imports
+    - pip install typer rich boto3
     # Install test dependencies
     - cd lib/idp_common_pkg && pip install -e ".[test]" && cd ../..
 

CHANGELOG.md

@@ -7,10 +7,72 @@ SPDX-License-Identifier: MIT-0
 
 ### Added
 
+- **Modern Python-Based Publishing System**
+  - **Complete Rewrite**: Replaced legacy `publish.sh` bash script (583 lines) with modern `publish.py` Python script (1,294 lines)
+  - **Enhanced User Experience**: Rich console interface with progress bars, spinners, and colored output using Rich library
+  - **Concurrent Processing**: Multi-threaded artifact building and uploading for significantly improved performance
+  - **Cross-Platform Support**: Native support for Linux, macOS, and Windows environments
+  - **Intelligent Caching**: Advanced checksum-based build optimization to skip unnecessary rebuilds
+  - **Robust Error Handling**: Comprehensive error handling with detailed error messages and recovery suggestions
+  - **Resource Management**: Automatic cleanup of temporary files and proper resource management
+  - **CLI Interface**: Modern command-line interface using Typer with clear help documentation and parameter validation
+  - **Key Features**:
+    - Parallel S3 uploads with thread-safe progress tracking
+    - CloudFormation template validation and processing
+    - Automatic bucket creation with proper permissions and lifecycle policies
+    - Build artifact optimization with dependency tracking
+    - Comprehensive logging and debugging capabilities
+    - Memory-efficient processing for large artifacts
+
+- **Comprehensive Unit Test Suite for Publishing System**
+  - **Extensive Test Coverage**: 1,621 lines of unit tests covering 95%+ of publish.py functionality
+  - **Testing Framework**: Uses pytest with proper unit test markers (`@pytest.mark.unit`) following project testing standards
+  - **Mock Integration**: Comprehensive mocking of AWS services (S3, CloudFormation) for isolated unit testing without external dependencies
+  - **Cross-Platform Testing**: Tests for Linux, macOS, and Windows-specific functionality and edge cases
+  - **Error Scenario Coverage**: Tests for network failures, permission errors, malformed templates, and concurrent access scenarios
+  - **Performance Testing**: Tests for concurrent operations, memory usage, and resource cleanup
+  - **Key Test Categories**:
+    - IDPPublisher initialization and configuration
+    - S3 operations (bucket creation, uploads, permissions)
+    - CloudFormation template processing and validation
+    - Build system integration and checksum validation
+    - Error handling and recovery mechanisms
+    - Concurrent publishing workflows and thread safety
+
+- **Windows Development Environment Setup**
+  - **Automated Setup Script**: New `scripts/dev_setup.bat` (570 lines) for complete Windows development environment configuration
+  - **Comprehensive Installation**: Automated installation of all required development tools and dependencies
+  - **Tool Installation**:
+    - AWS CLI with proper configuration
+    - Python 3.13 (required for compatibility with latest dependencies)
+    - Node.js for React UI development
+    - Git for version control
+    - Docker for containerized development
+    - AWS SAM CLI for serverless application development
+    - Python dependencies (boto3, numpy 2.3.2, typer, rich)
+  - **Environment Configuration**: Automatic AWS credentials setup and project cloning with dependency installation
+  - **Administrator Privileges**: Proper handling of Windows administrator requirements for system-level installations
+  - **Error Handling**: Comprehensive error checking and user guidance throughout the setup process
+
 - **OCR Service Default Image Sizing for Resource Optimization**
   - Implemented automatic default image size limits (951×1268) when no image sizing configuration is provided
   - **Key Benefits**: Reduction in vision model token consumption, prevents OutOfMemory errors during concurrent processing, improves processing speed and reduces bandwidth usage
 
+### Changed
+
+- **Publishing Workflow Modernization**
+  - Migrated from bash-based to Python-based publishing system for better maintainability and cross-platform support
+  - Improved build performance through intelligent caching and concurrent processing
+  - Enhanced developer experience with rich console output and clear progress indicators
+  - Better error diagnostics and troubleshooting capabilities
+
+### Technical Improvements
+
+- **Build System Optimization**: Checksum-based incremental builds reduce unnecessary processing time
+- **Memory Management**: Efficient handling of large artifacts and proper resource cleanup
+- **Thread Safety**: Concurrent operations with proper synchronization and error handling
+- **Code Quality**: Comprehensive unit testing ensures reliability and maintainability of the publishing system
+
 
 
 

docs/deployment.md

@@ -26,6 +26,7 @@ You need to have the following packages installed on your computer:
 3. [sam (AWS SAM)](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html)
 4. python 3.11 or later
 5. A local Docker daemon
+6. Python packages for publish.py: `pip install boto3 typer rich botocore`
 
 For guidance on setting up a development environment, see: [Development Environment Setup Guide on Linux](./setup-development-env-linux.md) or [Development Environment Setup Guide on MacOS](./setup-development-env-macos.md)
 
@@ -40,28 +41,53 @@ To build and publish your own template to your own S3 bucket:
 * `cfn_bucket_basename`: A prefix added to the beginning of the bucket name (e.g. `idp-1234567890` to ensure global uniqueness) 
 * `cfn_prefix`: A prefix added to CloudFormation resources (e.g. `idp` or `idp-dev`)
 
-Navigate into the project root directory and, in a bash shell, run:
+Navigate into the project root directory and run:
+
+#### Using publish.py (Recommended)
 
 ```bash
-./publish.sh <cfn_bucket_basename> <cfn_prefix> <region e.g. us-east-1>
+python3 publish.py <cfn_bucket_basename> <cfn_prefix> <region> [--verbose]
 ```
 
-This script:
-- Checks your system dependencies for required packages
-- Creates CloudFormation templates and asset zip files
-- Publishes the templates and required assets to an S3 bucket in your account
-- The bucket will be named `<cfn_bucket_basename>-<region>` (created if it doesn't exist)
+**Parameters:**
+- `cfn_bucket_basename`: A prefix for the S3 bucket name (e.g., `idp-1234567890`)
+- `cfn_prefix`: S3 prefix for artifacts (e.g., `idp`)
+- `region`: AWS region for deployment (e.g., `us-east-1`)
+- `--verbose` or `-v`: (Optional) Enable detailed error output for debugging build failures
 
 Example:
 ```bash
-./publish.sh idp-1234567890 idp us-east-1
+python3 publish.py idp-1234567890 idp us-east-1
+```
+
+**Troubleshooting Build Issues:**
+If the build fails, use the `--verbose` flag to see detailed error messages:
+```bash
+python3 publish.py idp-1234567890 idp us-east-1 --verbose
 ```
 
-Optional: Add a final parameter `public` if you want to make the published artifacts publicly accessible:
+This will show:
+- Exact SAM build commands being executed
+- Complete error output from failed builds
+- Python version compatibility issues
+- Missing dependencies or configuration problems
+
+#### Using publish.sh (Legacy)
+
 ```bash
-./publish.sh idp-1234567890 idp us-east-1 public
+./publish.sh <cfn_bucket_basename> <cfn_prefix> <region>
 ```
-Note: Your bucket and account must be configured not to Block Public Access using new ACLs.
+
+Example:
+```bash
+./publish.sh idp-1234567890 idp us-east-1
+```
+
+Both scripts:
+- Check your system dependencies for required packages
+- Create CloudFormation templates and asset zip files
+- Publish the templates and required assets to an S3 bucket in your account
+- The bucket will be named `<cfn_bucket_basename>-<region>` (created if it doesn't exist)
 
 When completed, the script displays:
 - The CloudFormation template's S3 URL

docs/setup-development-env-linux.md

@@ -112,13 +112,50 @@ Host genai-idp-dev
 ### Refer this link for AWS configure
 https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html  
 
-# Step 6: Run Publish Script 
-## Test Build Process
+# Step 6: Install Python Dependencies for publish.py
+
+## Install Required Python Packages
+```bash
+pip install boto3 typer rich botocore
+```
+
+# Step 7: Run Publish Script 
+
+## Using publish.py (Recommended)
+
+### Test publish script help
+```bash
+python3 publish.py --help
+```
+
+### Test build using publish.py
+```bash
+python3 publish.py bucket_name build-test us-east-1
+```
+
+### Troubleshooting Build Issues
+If the build fails, use the `--verbose` flag to see detailed error messages:
+```bash
+python3 publish.py bucket_name build-test us-east-1 --verbose
+```
+
+The verbose flag will show:
+- Exact SAM build commands being executed
+- Complete error output from failed builds
+- Python version compatibility issues
+- Missing dependencies or configuration problems
+
+## Using publish.sh (Legacy)
+
 ### Test publish script help
+```bash
 ./publish.sh --help
+```
 
-### Test build (this will take 10-15 minutes)
+### Test build using publish.sh
+```bash
 ./publish.sh bucket_name build-test us-east-1
+```
 
 # Step 7: Cline & Q Developer installation
 ### Cline

docs/setup-development-env-macos.md

@@ -242,10 +242,55 @@ EOF
 - SAM CLI: run `sam --version`.
 - Docker: launch Docker.app, then run `docker version` in Terminal.
 
-## **12) Configure AWS CLI**
+## **12) Install Python Dependencies for publish.py**
+
+Install the required Python packages for the publish.py script:
+
+```bash
+pip install boto3 typer rich botocore
+```
+
+## **13) Configure AWS CLI**
 ### Refer this link for AWS configure
 https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html  
-# Troubleshooting Tips
+
+## **14) Run Publish Script**
+
+### Using publish.py (Recommended)
+
+Test publish script help:
+```bash
+python3 publish.py --help
+```
+
+Test build using publish.py:
+```bash
+python3 publish.py bucket_name build-test us-east-1
+```
+
+**Troubleshooting Build Issues:**
+If the build fails, use the `--verbose` flag to see detailed error messages:
+```bash
+python3 publish.py bucket_name build-test us-east-1 --verbose
+```
+
+The verbose flag will show:
+- Exact SAM build commands being executed
+- Complete error output from failed builds
+- Python version compatibility issues
+- Missing dependencies or configuration problems
+
+### Using publish.sh (Legacy)
+
+Test publish script help:
+```bash
+./publish.sh --help
+```
+
+Test build using publish.sh:
+```bash
+./publish.sh bucket_name build-test us-east-1
+```
 
 - If `brew` is not found, add it to PATH: 
 ```bash
@@ -258,7 +303,7 @@ eval "$(/opt/homebrew/bin/brew shellenv 2>/dev/null || /usr/local/bin/brew shell
 
 - Docker needs to be running before `docker` commands succeed. Launch Docker.app and wait for the whale icon to be steady.
 
-## Cline & Q Developer installation
+# Troubleshooting Tips
 ### Cline
 What it is: An AI coding assistant that runs as a VS Code extension, powered by various LLMs (Claude, GPT, etc.)  
 

docs/troubleshooting.md

@@ -198,3 +198,53 @@ Test system capacity and identify bottlenecks:
 2. Start with low document rates and increase gradually
 3. Monitor CloudWatch metrics for saturation points
 4. Identify bottlenecks and optimize configuration
+
+## Build and Deployment Issues
+
+### Publishing Script Failures
+
+| Issue | Resolution |
+|-------|------------|
+| **Generic "Failed to build" error** | Use `--verbose` flag to see detailed error messages: `python3 publish.py bucket prefix region --verbose` |
+| **Python version mismatch** | Ensure Python 3.13 is installed and available in PATH. Check with `python3 --version` |
+| **SAM build fails** | Verify SAM CLI is installed and up to date. Check Docker is running if using containerized builds |
+| **Missing dependencies** | Install required packages: `pip install boto3 typer rich botocore` |
+| **Permission errors** | Verify AWS credentials are configured and have necessary S3/CloudFormation permissions |
+
+### Common Build Error Messages
+
+**Python Runtime Error:**
+```
+Error: PythonPipBuilder:Validation - Binary validation failed for python, searched for python in following locations: [...] which did not satisfy constraints for runtime: python3.13
+```
+**Resolution:** Install Python 3.13 and ensure it's in your PATH, or use the `--use-container` flag for containerized builds.
+
+**Docker Not Running:**
+```
+Error: Running AWS SAM projects locally requires Docker
+```
+**Resolution:** Start Docker daemon before running the publish script.
+
+**AWS Credentials Not Found:**
+```
+Error: Unable to locate credentials
+```
+**Resolution:** Configure AWS credentials using `aws configure` or set environment variables.
+
+### Verbose Mode Usage
+
+For detailed debugging information, always use the `--verbose` flag when troubleshooting build issues:
+
+```bash
+# Standard usage
+python3 publish.py my-bucket idp us-east-1
+
+# Verbose mode for troubleshooting
+python3 publish.py my-bucket idp us-east-1 --verbose
+```
+
+Verbose mode provides:
+- Exact SAM build commands being executed
+- Complete stdout/stderr from failed operations
+- Python environment and dependency information
+- Detailed error traces and stack traces