Skip to content

Conversation

@ethanndickson
Copy link
Member

@ethanndickson ethanndickson commented Dec 5, 2025

Stack:

  1. 🤖 feat: add UI styling for background bash processes #923

  2. 🤖 feat: add background bash process execution with SSH support #920 (base) <- This PR


Summary

Adds run_in_background=true option to the bash tool, enabling agents to spawn long-running processes (dev servers, builds, file watchers) that persist independently.

Why This Approach

We needed background process support that works identically for both local and SSH runtimes. A few considerations drove the design:

  1. PTY-based output doesn't fit remote use cases. For SSH, maintaining a persistent PTY connection just to let agents read stdout would be fragile and complex. Agents need to search, filter, and tail output—not consume a live stream.

  2. File-based output lets agents use standard tools. By writing stdout/stderr to files on the target machine, agents can use tail -f, grep, head, etc. to inspect output. We don't need to reimplement these filtering capabilities in our own tooling.

  3. A proper long-lived remote daemon is future work. Ideally, SSH remotes would have a persistent mux process (or agent binary) that manages background jobs directly. The user's frontend would just connect to it. That's a significant architectural change. This PR provides background shell support without requiring that investment—the file-based approach works today with no remote-side dependencies.

Architecture

AI Tools (bash, bash_background_list, bash_background_terminate)
    ↓
BackgroundProcessManager (lifecycle, in-memory tracking)
    ↓
Runtime.spawnBackground() (LocalBaseRuntime / SSHRuntime)
    ↓
BackgroundHandle (file-based output & status)
    ↓
backgroundCommands.ts (shared shell builders for Local/SSH parity)

Key Design Decisions

Decision Rationale
File-based output Works identically for local and SSH, agents read via tail/cat/grep.
set -m + nohup Robust process isolation, PID === PGID for clean group termination
Workspace-scoped Processes tied to workspace, cleaned up on workspace removal
Lazy status refresh No polling overhead, reads exit_code file on list()
Cleanup on compaction Background processes terminated before compaction, as they're not guaranteed to be included (Claude Code does this too)

Tools

  • bash(run_in_background=true) — spawns process, returns stdout_path/stderr_path
  • bash_background_list — lists processes with status and file paths
  • bash_background_terminate — kills process group (SIGTERM → wait → SIGKILL)

Output Structure

/tmp/mux-bashes/{workspaceId}/{bg-xxx}/
├── stdout.log    # Process stdout
├── stderr.log    # Process stderr  
├── exit_code     # Written by trap on exit
└── meta.json     # Process metadata

Technical Details

Process Spawning

The spawn command uses a subshell with job control enabled:

(set -m; nohup bash -c 'WRAPPER_SCRIPT' > stdout.log 2> stderr.log < /dev/null & echo $!)

Key elements:

  • set -m — Enables bash job control, which makes backgrounded processes become their own process group leader (PID === PGID). This is a bash builtin available on all platforms.
  • nohup — Prevents SIGHUP from killing the process when the terminal closes.
  • Subshell (...) — Isolates the process group so the outer shell exits immediately after echoing the PID.
  • < /dev/null — Detaches stdin so the process doesn't block waiting for input.

Exit Code Detection

The wrapper script sets up a trap to capture the exit code:

trap 'echo $? > exit_code' EXIT && cd /path && export ENV=val && USER_SCRIPT

When the process exits (normally or via signal), the trap writes $? to the exit_code file. Mux reads this file to determine if the process is still running (exit_code doesn't exist) or has exited (file contains the code).

Process Group Termination

Because set -m ensures PID === PGID, we can kill the entire process tree using a negative PID:

kill -15 -PID; sleep 2; if kill -0 -PID; then kill -9 -PID; echo 137 > exit_code; else echo 143 > exit_code; fi

Sequence:

  1. Send SIGTERM (-15) to the process group (-PID targets the group)
  2. Wait 2 seconds for graceful shutdown
  3. Check if any process in the group survives (kill -0 -PID)
  4. If still alive, send SIGKILL (-9) and record exit code 137
  5. Otherwise, record exit code 143 (SIGTERM)

This ensures child processes spawned by the background job are also terminated, preventing orphaned processes.

Cross-Platform Compatibility

Feature Linux macOS Windows (MSYS2)
set -m ✓ bash builtin ✓ bash builtin ✓ bash builtin
kill -15/-9 -PID
nohup
Path format POSIX POSIX Converted via cygpath

Using set -m instead of platform-specific tools like setsid (Linux-only) ensures the same code works everywhere.

Platform Support

  • Linux/macOS/Windows MSYS2: set -m + nohup pattern (universal)
  • SSH: Same pattern executed remotely

Testing

  • 20 unit tests in backgroundProcessManager.test.ts (including process group termination)
  • 7 unit tests for background execution in bash.test.ts
  • 6 unit tests in bash_background_list.test.ts (including display_name)
  • 5 unit tests in bash_background_terminate.test.ts
  • 19 unit tests in backgroundCommands.test.ts
  • 7 runtime tests in tests/runtime/runtime.test.ts (Local & SSH)
  • 3 end-to-end integration tests in tests/ipc/backgroundBash.test.ts (real AI calls)

Manual Testing

Tested on Linux, macOS, and Windows (MSYS2) to verify:

  • Background processes spawn correctly
  • Process trees are fully terminated (parent and children)
  • Output files are written and readable

Generated with mux

@ethanndickson ethanndickson force-pushed the background-process-file-storage branch 2 times, most recently from 0c99075 to 92ef0c8 Compare December 5, 2025 01:09
@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson ethanndickson force-pushed the background-process-file-storage branch from abc50fe to 99af592 Compare December 5, 2025 03:42
@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson
Copy link
Member Author

@codex review

@ethanndickson ethanndickson force-pushed the background-process-file-storage branch from 36ea849 to c14210c Compare December 5, 2025 05:01
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@ethanndickson ethanndickson force-pushed the background-process-file-storage branch from 228b9dd to 6d815be Compare December 8, 2025 02:36
The bash tool blocks commands starting with 'sleep' to prevent wasting
time. Changed test commands from 'sleep 30' to 'true && sleep 30' to
bypass this validation while still testing background process behavior.
@ethanndickson ethanndickson force-pushed the background-process-file-storage branch from b2e8055 to 211d10c Compare December 8, 2025 03:14
bash_background_terminate returns { message: ... } not { output: ... }
- Replace platform-specific PGID detection with universal fallback chain:
  ps -o pgid= → /proc/$!/pgid → PID (works on Linux, macOS, MSYS2)
- Use numeric signals (-15, -9) instead of named (-TERM, -KILL) for MSYS2 compat
- Extract parsePidPgid() helper to backgroundCommands.ts (DRY)
- Simplify LocalBackgroundHandle.terminate() from ~90 to ~17 lines
  - Remove Windows-specific conditional code
  - Now uses buildTerminateCommand via bash shell (same as SSH)
- Add PGID parameter to SSHBackgroundHandle for proper process group termination

_Generated with mux_
Without job control, backgrounded processes inherit the parent's PGID.
On Linux locally, this caused 'kill -PGID' to kill mux itself.

Adding 'set -m' enables bash job control, which creates a new process
group for backgrounded processes (PID === PGID), making it safe to kill.

_Generated with mux_
With set -m, PID === PGID is guaranteed (process is its own group leader).
Remove the redundant PGID lookup chain (ps → /proc → fallback) and use
just the PID throughout.

- buildSpawnCommand: echo $! instead of PGID lookup
- parsePidPgid → parsePid: return number instead of {pid, pgid}
- Remove pgid field from LocalBackgroundHandle and SSHBackgroundHandle
- Clean up tests: 46 → 20 tests, remove redundant assertions
@ethanndickson ethanndickson force-pushed the background-process-file-storage branch from dacdda9 to df15f39 Compare December 8, 2025 05:17
The EXIT trap in the process writes $? (which is 0 for SIGTERM) and would
overwrite our exit code if we wrote it immediately. Now we wait for the
process to exit before writing 143 (SIGTERM) or 137 (SIGKILL).
kill -0 ${pid} only checks if the parent is alive, but children may
survive SIGTERM. Changed to kill -0 ${negPid} to check if any process
in the group is still alive before deciding to send SIGKILL.
@ethanndickson ethanndickson requested a review from Copilot December 8, 2025 06:26

This comment was marked as resolved.

@ethanndickson ethanndickson requested a review from Copilot December 8, 2025 06:44
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 43 out of 45 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Use execFileSync with args array in toPosixPath to avoid shell injection
- Add 5s timeout to app dispose to ensure quit even if disposal hangs
- Log warning when SSH /home/ethan resolution fails and falls back to /tmp
- Add random suffix to test markers to prevent collision
@ethanndickson ethanndickson force-pushed the background-process-file-storage branch 2 times, most recently from 8064cbd to 54cd503 Compare December 8, 2025 07:05
- Test that child processes are terminated when parent is killed (validates set -m)
- Test that display_name round-trips through spawn and list
@ethanndickson ethanndickson force-pushed the background-process-file-storage branch from 54cd503 to badae77 Compare December 8, 2025 07:11
@ethanndickson ethanndickson added this pull request to the merge queue Dec 8, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 8, 2025
@ethanndickson ethanndickson added this pull request to the merge queue Dec 8, 2025
Merged via the queue into main with commit acb9052 Dec 8, 2025
19 checks passed
@ethanndickson ethanndickson deleted the background-process-file-storage branch December 8, 2025 07:28
github-merge-queue bot pushed a commit that referenced this pull request Dec 8, 2025
### Stack:

1. #923 <- This PR

1. #920 (base)



---



Adds custom UI components for background bash tools:



- **BashToolCall**: Shows `⚡ background • display_name` for background
spawns instead of timeout/duration; shows output file paths in expanded
view

- **BashBackgroundListToolCall**: New component showing process list
with status badges, exit codes, uptimes, scripts, and output file paths

- **BashBackgroundTerminateToolCall**: New component showing terminated
process with display_name from result



<img width="984" height="946" alt="image"
src="https://github.com/user-attachments/assets/3de5a0bb-6c00-4641-96cb-65c67e3d0085"
/>



**Testing:** Manually tested on Linux, Mac, and Windows.

Closes #493 


_Generated with mux_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant