From d48cc4fb547322c692aeb1e300027f2e8a9cfaa0 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Tue, 6 Jan 2026 19:03:49 +1100 Subject: [PATCH] feat: add openspec proposal for concurrent vscode instances --- .../add-vscode-concurrent-instances/design.md | 33 +++++++++++++++++++ .../proposal.md | 15 +++++++++ .../specs/eval-cli/spec.md | 28 ++++++++++++++++ .../specs/evaluation/spec.md | 29 ++++++++++++++++ .../add-vscode-concurrent-instances/tasks.md | 15 +++++++++ 5 files changed, 120 insertions(+) create mode 100644 openspec/changes/add-vscode-concurrent-instances/design.md create mode 100644 openspec/changes/add-vscode-concurrent-instances/proposal.md create mode 100644 openspec/changes/add-vscode-concurrent-instances/specs/eval-cli/spec.md create mode 100644 openspec/changes/add-vscode-concurrent-instances/specs/evaluation/spec.md create mode 100644 openspec/changes/add-vscode-concurrent-instances/tasks.md diff --git a/openspec/changes/add-vscode-concurrent-instances/design.md b/openspec/changes/add-vscode-concurrent-instances/design.md new file mode 100644 index 00000000..1e4eaba5 --- /dev/null +++ b/openspec/changes/add-vscode-concurrent-instances/design.md @@ -0,0 +1,33 @@ +## Context +VS Code subagent dispatch uses the VS Code CLI and focuses a workspace before sending chat instructions. With multiple VS Code windows open, the chat command can attach to the wrong window if focus changes between the focus step and the CLI dispatch. This forces the CLI to cap VS Code workers at 1. + +## Goals / Non-Goals +- Goals: + - Enable safe parallel VS Code eval workers by routing each request to a deterministic VS Code instance. + - Preserve current default behavior and remain non-breaking. + - Keep configuration minimal and opt-in. +- Non-Goals: + - Redesign subagent locking or workspace provisioning. + - Change VS Code prompt structure or Copilot behavior. + +## Decisions +- Decision: Introduce an opt-in `vscode_instance_mode: isolated` target setting. + - When enabled, the provider launches and targets per-instance VS Code processes using unique `--user-data-dir` and `--extensions-dir` paths. + - Default remains focus-based with a worker cap. +- Decision: Add `vscode_instance_root` (optional) to control where per-instance data directories are created; default under `.agentv/vscode/instances/`. +- Decision: Add `vscode_instance_count` (optional) to control the size of the instance pool; default to the eval run max concurrency when available. +- Decision: Pass VS Code CLI arguments through subagent (new `vscodeArgs` option) instead of embedding them in `vscode_cmd` strings. + +## Risks / Trade-offs +- Extra disk usage and startup time per isolated instance (extensions and user data). +- Requires a subagent version that supports forwarding VS Code CLI arguments; older versions will not work with isolation. +- OS-specific path/quoting behavior must be validated, especially on Windows. + +## Migration Plan +- No migration required. Existing targets keep `vscode_instance_mode` unset and continue to run with a single worker. +- Users who want parallel VS Code evals set `vscode_instance_mode: isolated` and optionally `vscode_instance_root`. + +## Open Questions +- Should the instance count be explicitly configurable, or always match resolved worker count? +- Should extensions be shared across instances (single extensions dir) to reduce startup time? +- Do we need additional CLI flags to override instance root on a per-run basis? diff --git a/openspec/changes/add-vscode-concurrent-instances/proposal.md b/openspec/changes/add-vscode-concurrent-instances/proposal.md new file mode 100644 index 00000000..320c7ac4 --- /dev/null +++ b/openspec/changes/add-vscode-concurrent-instances/proposal.md @@ -0,0 +1,15 @@ +# Change: Concurrent VS Code instances for eval workers + +## Why +The VS Code provider currently forces a single worker because chat dispatch targets the last focused VS Code window, which can route prompts to the wrong workspace when multiple instances are open. This makes parallel evals unreliable and forces manual tracking of subagent workspaces. + +## What Changes +- Add an opt-in VS Code target mode that launches isolated VS Code instances per worker to avoid focus races, with optional instance count control. +- Allow the eval CLI to run multiple workers for VS Code targets when isolation is enabled. +- Extend VS Code provider configuration to derive per-instance CLI arguments (user-data/extension dirs) and track instance usage. +- Update the subagent integration to accept VS Code CLI arguments for deterministic instance routing. + +## Impact +- Affected specs: `openspec/specs/eval-cli/spec.md`, `openspec/specs/evaluation/spec.md` +- Affected code: `apps/cli/src/commands/eval/run-eval.ts`, `packages/core/src/evaluation/providers/vscode.ts`, `packages/core/src/evaluation/providers/targets.ts`, subagent integration +- Docs/templates: VS Code target docs and templates (skills/references) describing the new isolation mode diff --git a/openspec/changes/add-vscode-concurrent-instances/specs/eval-cli/spec.md b/openspec/changes/add-vscode-concurrent-instances/specs/eval-cli/spec.md new file mode 100644 index 00000000..fb3a6039 --- /dev/null +++ b/openspec/changes/add-vscode-concurrent-instances/specs/eval-cli/spec.md @@ -0,0 +1,28 @@ +## MODIFIED Requirements +### Requirement: Concurrency and Worker Scheduling + +The CLI SHALL schedule work across eval files and cases within configured worker limits. + +#### Scenario: Worker limits and validation + +- **WHEN** the user provides `--workers ` +- **THEN** the CLI enforces a minimum of 1 and maximum of 50 workers (erroring when out of range) +- **AND** defaults to 1 worker when the flag is omitted or non-numeric + +#### Scenario: File-level worker balancing + +- **WHEN** multiple eval files are provided with `--workers N` +- **THEN** the CLI assigns at least one worker per file up to `N`, dividing remaining workers evenly per file to cap in-flight evals + +#### Scenario: VS Code worker cap (focused mode) + +- **WHEN** the selected target is `vscode`/`vscode-insiders` +- **AND** the target does NOT set `vscode_instance_mode: isolated` +- **THEN** the CLI limits workers to 1 (with a warning when a higher value was requested) to avoid window-focus contention + +#### Scenario: VS Code isolation enables concurrency + +- **WHEN** the selected target is `vscode`/`vscode-insiders` +- **AND** the target sets `vscode_instance_mode: isolated` +- **THEN** the CLI allows the configured worker count within bounds +- **AND** provisions a matching number of subagents before execution diff --git a/openspec/changes/add-vscode-concurrent-instances/specs/evaluation/spec.md b/openspec/changes/add-vscode-concurrent-instances/specs/evaluation/spec.md new file mode 100644 index 00000000..ef109d5c --- /dev/null +++ b/openspec/changes/add-vscode-concurrent-instances/specs/evaluation/spec.md @@ -0,0 +1,29 @@ +## ADDED Requirements +### Requirement: VS Code isolated instance mode +The system SHALL support an opt-in isolated instance mode for the VS Code provider to avoid focus-based routing. Targets MAY configure `vscode_instance_mode`, `vscode_instance_root`, and `vscode_instance_count` to control this behavior. + +#### Scenario: Enable isolated instances +- **GIVEN** a target uses `provider: vscode` (or `vscode-insiders`) +- **AND** the target sets `vscode_instance_mode: isolated` +- **WHEN** the provider dispatches a request +- **THEN** it assigns the request to a distinct VS Code instance up to the configured instance count +- **AND** launches the instance with unique `--user-data-dir` and `--extensions-dir` paths +- **AND** passes those instance arguments on every VS Code CLI invocation for that request + +#### Scenario: Default focused mode +- **GIVEN** a target uses `provider: vscode` (or `vscode-insiders`) +- **AND** `vscode_instance_mode` is unset or `focused` +- **WHEN** the provider dispatches a request +- **THEN** it uses the existing focus-based dispatch behavior without instance isolation + +#### Scenario: Instance root resolution +- **GIVEN** a target sets `vscode_instance_root` +- **WHEN** isolated instances are created +- **THEN** instance data directories are created under that root with stable per-instance subfolders (e.g., `instance-1`, `instance-2`) + +#### Scenario: Instance count default +- **GIVEN** a target sets `vscode_instance_mode: isolated` +- **AND** `vscode_instance_count` is not specified +- **WHEN** the eval run provides a max concurrency value +- **THEN** the system uses that max concurrency as the instance count +- **AND** falls back to 1 when no max concurrency is available diff --git a/openspec/changes/add-vscode-concurrent-instances/tasks.md b/openspec/changes/add-vscode-concurrent-instances/tasks.md new file mode 100644 index 00000000..d50e3bd1 --- /dev/null +++ b/openspec/changes/add-vscode-concurrent-instances/tasks.md @@ -0,0 +1,15 @@ +## 1. Implementation +- [ ] 1.1 Extend VS Code target schema to accept `vscode_instance_mode`, `vscode_instance_root` (optional), and `vscode_instance_count` (optional) and resolve env vars +- [ ] 1.2 Update VS Code provider to derive per-instance CLI args (`--user-data-dir`, `--extensions-dir`) and pin requests to an instance when isolation is enabled +- [ ] 1.3 Update CLI worker resolution to allow >1 workers for VS Code targets when isolation is enabled and to provision matching subagents +- [ ] 1.4 Add/adjust unit tests for VS Code worker limits and new isolation settings +- [ ] 1.5 Update template docs in `apps/cli/src/templates/.claude/skills/agentv-eval-builder/` to document isolation mode + +## 2. Dependencies +- [ ] 2.1 Update subagent integration to accept VS Code CLI arguments for deterministic instance targeting + +## 3. Verification +- [ ] 3.1 Run `bun run build` +- [ ] 3.2 Run `bun run typecheck` +- [ ] 3.3 Run `bun run lint` +- [ ] 3.4 Run `bun test`