🩹 [Patch]: Consolidate test setup/teardown tasks

[MariusStorhaug]

Describe the change

Now that Process-PSModule supports BeforeAll and AfterAll scripts, that run once before and after all the parallel tests, we should move some of the setup/teardown tasks to these. This will hopefully speed up the testing process, and result in fewer failures due to ratelimiting.

Problem

The primary bottleneck is secondary rate limits on repository creation. Multiple test files independently create their own repositories per auth case, resulting in a large burst of New-GitHubRepository calls that trigger GitHub's secondary rate limits.

Current repo creation per auth case (5 active cases)

Test File	Repos Created	Other Resources
Repositories.Tests	7	5 teams (org auth), forks, templates
Secrets.Tests	3	Org secrets, repo secrets, environment + env secrets
Variables.Tests	3	Org variables, repo variables, environment + env variables
Releases.Tests	1	5 releases, 6+ release assets
Environments.Tests	0 (uses shared)	Environments
Organizations.Tests	0	1 org (enterprise only), app installations, invitations
Teams.Tests	0	6 teams (org only)
Total per auth case	~14
Total across 5 cases	~70

On top of that, BeforeAll.ps1 creates 1 repo per case (5 total). So a full test run attempts ~75 repository creations plus cleanup.

Current Test Structure

Each test file currently performs repeated setup/teardown in individual Context blocks:

• Creates unique test resources (repos, teams, orgs) per Context
• Connects/disconnects GitHub accounts multiple times
• Cleans up resources after each Context

Proposed Changes

tests/BeforeAll.ps1 — Shared Infrastructure Setup

Create shared test infrastructure that ALL tests can use. For each auth case:

1. Cleanup stale resources from previous failed runs (repos, teams, org secrets/variables matching test prefixes)
2. Create a pool of shared repositories — enough to cover the needs of all test files:
   • 1 base repo per case (already done in PR 🩹 [Patch]: Consolidate test setup/teardown tasks #541) — used by Environments, general tests
   • 3 additional repos per case for Secrets/Variables selected-repository testing
   • 1 repo per case pre-populated with tags/README for Releases testing
   • Repos needed by Repositories.Tests for settings, permissions, etc.
3. Create shared teams (org auth cases only) — admin, maintain, push, triage, pull
4. Create shared environments on the base repo for Secrets/Variables environment-scope testing

Important: This does NOT replace the BeforeAll blocks in test files, which are still needed for:

• Authentication (Connect-GitHubAccount, Connect-GitHubApp)
• Context-specific setup
• Loading test data (AuthCases.ps1)

tests/AfterAll.ps1 — Shared Infrastructure Teardown

Clean up all shared test infrastructure. For each auth case:

1. Remove org-level secrets and variables matching test prefixes
2. Remove teams matching test prefixes
3. Remove all test repositories matching test prefixes
4. Remove test organizations (enterprise auth only)

Test Files — Consume Shared Infrastructure

Test file Context blocks should continue to:

• BeforeAll: Connect to GitHub with appropriate credentials
• AfterAll: Disconnect GitHub sessions
• Use pre-created infrastructure via Get-GitHubRepository instead of New-GitHubRepository

Test Files by Migration Complexity

Lightweight (read-only or minimal changes)

These files create no resources or only read-only API calls:

• GitHubFormatter.Tests — Pure unit tests, no API calls. No consolidation needed.
• Emojis.Tests — Read-only (downloads emojis). No consolidation needed.
• Enterprise.Tests — Read-only (Get-GitHubEnterprise). No consolidation needed.
• Permissions.Tests — Read-only (permission catalog checks). No consolidation needed.
• Apps.Tests — Mostly read-only; creates installation access tokens but no persistent resources.
• Artifacts.Tests — Read-only (reads artifacts from current workflow run).
• GitHub.Tests — Mostly read-only (module config, meta endpoints, rate limits, viewer, status). No repos created.
• Users.Tests — Creates/removes user emails inline. User-specific, cannot be pre-shared.

Medium (use shared repos, remove per-test repo creation)

• Environments.Tests — Already migrated in PR 🩹 [Patch]: Consolidate test setup/teardown tasks #541. Uses shared repo for environment CRUD.
• Teams.Tests — Creates 6 teams per org auth case. Could use shared teams or continue creating test-specific teams (team creation is not a rate-limit hotspot).

Heavy (significant restructuring needed)

• Releases.Tests — Creates 1 repo + 5 releases + 6 assets per case. Needs a dedicated pre-created repo with tags.
• Secrets.Tests — Creates 3 repos + 1 environment per case. Needs shared repo pool + shared environment.
• Variables.Tests — Creates 3 repos + 1 environment per case (nearly identical to Secrets). Same shared infrastructure.
• Repositories.Tests — Heaviest: 7 repos + 5 teams per case. Tests repo creation, settings, templates, forks, permissions. Some tests inherently NEED to create repos (e.g., New-GitHubRepository, fork, template tests). Strategy: pre-create repos that are only read/updated, keep creation tests that specifically test New-GitHubRepository.
• Organizations.Tests — Creates real organizations under msx enterprise + app installations. Enterprise-specific, likely needs to keep its own setup but with better stale cleanup.

Special Patterns to Address

Fork and Template repos (Repositories.Tests)

• Uses PSModule/Template-Action and PSModule/Template-Docs as template sources (external, not created)
• Uses psmodule-test/fork-{OS} as fork source
• Fork and template repo creation tests inherently need to call New-GitHubRepository — these cannot be fully consolidated

Selected-repository testing (Secrets.Tests, Variables.Tests)

• Both need 3 repos to test Add/Remove/Set-GitHub{Secret,Variable}SelectedRepository
• These 3 repos could be shared between Secrets and Variables tests if created in BeforeAll

Organization lifecycle (Organizations.Tests)

• Creates orgs under the msx enterprise, installs apps, sends invitations
• This is inherently a lifecycle test — cannot pre-create the orgs

Benefits

1. Speed: Infrastructure created once instead of ~75 times
2. Rate Limiting: Dramatically fewer New-GitHubRepository calls — the primary secondary rate limit trigger
3. Reliability: Consistent test infrastructure; fewer transient failures from rate limits
4. Cost: Fewer parallel operations hitting API limits
5. Maintainability: Centralized infrastructure definitions; test files focus on behavior

Implementation Strategy

1. ✅ Create tests/BeforeAll.ps1 with infrastructure provisioning (done in PR 🩹 [Patch]: Consolidate test setup/teardown tasks #541 for base repos)
2. ✅ Create tests/AfterAll.ps1 with cleanup logic (done in PR 🩹 [Patch]: Consolidate test setup/teardown tasks #541)
3. ✅ Migrate Environments.Tests.ps1 as proof-of-concept (done in PR 🩹 [Patch]: Consolidate test setup/teardown tasks #541)
4. Expand BeforeAll to create the full shared repo pool (base + extras for Secrets/Variables/Releases)
5. Expand BeforeAll to create shared teams and environments
6. Expand AfterAll to clean up org secrets, variables, and teams
7. Migrate lightweight test files (move back from tmp/, no structural changes needed)
8. Migrate medium-complexity test files (Teams, Releases)
9. Migrate heavy test files (Secrets, Variables, Repositories)
10. Handle Organizations.Tests separately (enterprise-specific lifecycle)

Test Files Affected

All 15 test files will need updates to use shared infrastructure:

• Apps.Tests.ps1
• Artifacts.Tests.ps1
• Emojis.Tests.ps1
• Enterprise.Tests.ps1
• Environments.Tests.ps1
• GitHub.Tests.ps1
• GitHubFormatter.Tests.ps1
• Organizations.Tests.ps1
• Permissions.Tests.ps1
• Releases.Tests.ps1
• Repositories.Tests.ps1
• Secrets.Tests.ps1
• Teams.Tests.ps1
• Users.Tests.ps1
• Variables.Tests.ps1