Add PyRIT 0.10.0 technical specification and progress tracking for red team module

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/progress.md

@@ -0,0 +1,252 @@
+# PyRIT FoundryScenario Integration - Progress Tracking
+
+**Last Updated:** 2025-12-18  
+**Current Phase:** Planning Complete  
+**Next Milestone:** Phase 1 Implementation Start
+
+## Current Status
+
+### Overall Progress: 0% Complete
+
+| Phase | Status | Start Date | Completion Date | Progress |
+|-------|--------|------------|-----------------|----------|
+| Planning & Design | ✅ Complete | 2025-10-01 | 2025-12-18 | 100% |
+| Phase 1: Core Infrastructure | ⏳ Not Started | TBD | TBD | 0% |
+| Phase 2: Result Processing | ⏳ Not Started | TBD | TBD | 0% |
+| Phase 3: End-to-End Integration | ⏳ Not Started | TBD | TBD | 0% |
+| Phase 4: Testing & Documentation | ⏳ Not Started | TBD | TBD | 0% |
+
+## Blockers
+
+### High Priority
+
+1. **PyRIT 0.10.0 Upgrade Required:**
+   - **Issue:** Project currently depends on PyRIT version that may not include 0.10.0 features
+   - **Action:** Upgrade to PyRIT 0.10.0+ in requirements
+   - **Owner:** TBD
+   - **Status:** Not Started
+
+2. **SQLite Migration Path:**
+   - **Issue:** Need to ensure smooth transition from any existing DuckDB usage
+   - **Action:** Audit codebase for DuckDB references
+   - **Owner:** TBD
+   - **Status:** Not Started
+
+3. **RAI Service Endpoint Availability:**
+   - **Issue:** Need to validate RAI simulation endpoint is accessible and configured
+   - **Action:** Verify endpoint credentials and permissions
+   - **Owner:** TBD
+   - **Status:** Not Started
+
+4. **Breaking Change in PyRIT 0.10.0:**
+   - **Issue:** Current code uses `initialize_pyrit(memory_db_type=DUCK_DB)` which no longer exists
+   - **Location:** `_red_team.py` line 222
+   - **Fix Required:** Change to `initialize_pyrit(memory_db_type=SQLITE, memory_db_path=db_path)`
+   - **Impact:** High - must be addressed before Phase 1 implementation
+
+### Medium Priority
+
+5. **Test Infrastructure Setup:**
+   - **Issue:** Need mock PyRIT scenario for testing without external dependencies
+   - **Action:** Create test fixtures and mocks
+   - **Owner:** TBD
+   - **Status:** Not Started
+
+6. **Performance Baseline:**
+   - **Issue:** Need to establish current performance metrics before migration
+   - **Action:** Run performance benchmarks on existing implementation
+   - **Owner:** TBD
+   - **Status:** Not Started
+
+## Phase 1: Core Infrastructure (Not Started)
+
+### Tasks
+
+- [ ] Create strategy mapping module
+  - [ ] Define `ATTACK_STRATEGY_TO_FOUNDRY_STRATEGY` mapping
+  - [ ] Add unit tests for mapping correctness
+  - [ ] Document strategy equivalence
+
+- [ ] Update PyRIT initialization
+  - [ ] Fix breaking change in `_red_team.py` line ~234
+  - [ ] Implement SQLite database path configuration
+  - [ ] Add error handling for initialization failures
+  - [ ] Add logging for initialization steps
+
+- [ ] Implement scenario manager
+  - [ ] Create `_scenario_manager.py` module
+  - [ ] Implement FoundryScenario creation logic
+  - [ ] Add memory label configuration
+  - [ ] Implement scenario execution orchestration
+
+- [ ] Add context preservation
+  - [ ] Design memory label schema
+  - [ ] Implement label attachment during scenario creation
+  - [ ] Add label-based retrieval methods
+  - [ ] Test label persistence and retrieval
+
+### Deliverables
+
+- [ ] `_utils/strategy_mapping.py` with complete mappings
+- [ ] Updated `_red_team.py` with SQLite initialization
+- [ ] `_scenario_manager.py` with basic scenario orchestration
+- [ ] Unit tests with >80% coverage for new modules
+
+## Phase 2: Result Processing (Not Started)
+
+### Tasks
+
+- [ ] Create result converter module
+  - [ ] Implement `_result_converter.py`
+  - [ ] Use `get_message_pieces()` API
+  - [ ] Extract MessagePiece data correctly
+  - [ ] Handle edge cases (empty results, errors)
+
+- [ ] Update result processor
+  - [ ] Migrate from PromptRequestPiece to MessagePiece
+  - [ ] Update data access patterns
+  - [ ] Preserve existing result format
+  - [ ] Add backward compatibility checks
+
+- [ ] Integration with evaluation pipeline
+  - [ ] Connect result converter to evaluation processor
+  - [ ] Validate result schema compatibility
+  - [ ] Add result export functionality
+  - [ ] Test end-to-end result flow
+
+### Deliverables
+
+- [ ] `_result_converter.py` with full conversion logic
+- [ ] Updated `_result_processor.py` using MessagePiece
+- [ ] Integration tests for result processing
+- [ ] Documentation for result schema
+
+## Phase 3: End-to-End Integration (Not Started)
+
+### Tasks
+
+- [ ] Connect all components
+  - [ ] Wire scenario manager into `_red_team.py`
+  - [ ] Connect result converter to main flow
+  - [ ] Add orchestration logic
+  - [ ] Implement cleanup procedures
+
+- [ ] Error handling and resilience
+  - [ ] Add retry logic for transient failures
+  - [ ] Implement proper error propagation
+  - [ ] Add logging and diagnostics
+  - [ ] Handle partial success scenarios
+
+- [ ] Progress tracking
+  - [ ] Implement progress callbacks
+  - [ ] Add status reporting
+  - [ ] Create progress persistence
+  - [ ] Add cancellation support
+
+### Deliverables
+
+- [ ] Fully integrated red team scan functionality
+- [ ] Comprehensive error handling
+- [ ] Progress tracking implementation
+- [ ] Integration tests covering all strategies
+
+## Phase 4: Testing & Documentation (Not Started)
+
+### Tasks
+
+- [ ] Unit testing
+  - [ ] Achieve >90% coverage for new modules
+  - [ ] Add edge case tests
+  - [ ] Add error scenario tests
+  - [ ] Add performance tests
+
+- [ ] Integration testing
+  - [ ] End-to-end scan tests
+  - [ ] Multi-strategy tests
+  - [ ] Context preservation tests
+  - [ ] Backward compatibility tests
+
+- [ ] Documentation
+  - [ ] Update API documentation
+  - [ ] Create migration guide
+  - [ ] Add code examples
+  - [ ] Create sample notebooks
+
+### Deliverables
+
+- [ ] Test suite with >90% coverage
+- [ ] Published API documentation
+- [ ] Migration guide for users
+- [ ] Sample code and notebooks
+
+## Design Decisions
+
+| Decision | Rationale | Date |
+|----------|-----------|------|
+| Target PyRIT 0.10.0+ | Latest stable version with SQLite-only backend | 2025-12-18 |
+| Use SQLite memory | Only option in PyRIT 0.10.0+ (DuckDB removed) | 2025-12-18 |
+| Use MessagePiece data model | PyRIT 0.10.0 renamed PromptRequestPiece | 2025-12-18 |
+| Preserve public API | Ensure backward compatibility for users | 2025-11-15 |
+| Use memory labels for context | Enable filtering and reconstruction of scan sessions | 2025-11-15 |
+| Abstract FoundryStrategy mapping | Decouple Azure AI Evaluation from PyRIT internals | 2025-10-15 |
+| Maintain abstraction layer | Protect against future PyRIT breaking changes | 2025-10-15 |
+
+## Risk Register
+
+| Risk | Status | Mitigation |
+|------|--------|------------|
+| PyRIT API changes in 0.10.0 | ⚠️ Active | Documented in spec, ready to implement | 
+| SQLite performance at scale | 🔍 Monitoring | Will benchmark during Phase 2 |
+| Memory label key collisions | ✅ Mitigated | Use namespaced keys |
+| Backward compatibility issues | 🔍 Monitoring | Extensive testing planned in Phase 4 |
+
+## Metrics
+
+### Code Quality Targets
+- Test Coverage: >90%
+- Pylint Score: >9.0
+- Type Coverage: >95%
+- Documentation Coverage: 100%
+
+### Performance Targets
+- Scenario Execution Latency: <5% increase vs current
+- Memory Query Performance: <100ms for typical scan
+- Result Processing Throughput: >100 conversations/sec
+- Resource Utilization: <10% increase in memory/CPU
+
+## Team Communication
+
+### Weekly Sync Topics
+1. Blocker review and resolution
+2. Phase progress updates
+3. Design decision review
+4. Risk assessment
+5. Next week planning
+
+### Stakeholder Updates
+- **Weekly:** Progress summary to team leads
+- **Bi-weekly:** Demo to product management
+- **Monthly:** Executive summary with metrics
+
+## Next Actions
+
+1. **Immediate (This Week):**
+   - Assign owners to Phase 1 tasks
+   - Upgrade PyRIT to 0.10.0
+   - Set up development environment
+
+2. **Short Term (Next 2 Weeks):**
+   - Begin Phase 1 implementation
+   - Create strategy mapping module
+   - Fix breaking change in `_red_team.py`
+
+3. **Medium Term (Next Month):**
+   - Complete Phase 1
+   - Begin Phase 2
+   - Conduct first integration tests
+
+---
+
+**Document Version History:**
+- v2.0 (2025-12-18): Updated for PyRIT 0.10.0 alignment
+- v1.0 (2025-10-01): Initial progress tracking document