Osworld support #5

yidong72 · 2025-10-25T20:40:07Z

This change is worth documenting at https://docs.all-hands.dev/
Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

End-user friendly description of the problem this fixes or functionality this introduces.

Summarize what the PR does, explaining any non-trivial design decisions.

Link of any specific issues this addresses:

Copilot

Pull Request Overview

This PR adds OSWorld support by introducing OS-level interactions (computer_use tool) and updating the default base image from nikolaik/python-nodejs:python3.12-nodejs22 to ubuntu:24.04. The changes enable GUI automation capabilities through PyAutoGUI and Xvfb, along with supporting infrastructure for virtual display management.

Key changes:

Added OSInteractiveAction for OS-level interactions via PyAutoGUI
Changed default base container image to ubuntu:24.04 across all runtime configurations
Implemented virtual display (Xvfb) support with dynamic display number allocation

Reviewed Changes

Copilot reviewed 38 out of 39 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
openhands/events/action/os.py	New action type for OS interactions
openhands/agenthub/codeact_agent/tools/computer_use.py	Computer use tool definition with PyAutoGUI functions
openhands/runtime/action_execution_server.py	Xvfb initialization and OS action execution
openhands/runtime/utils/system.py	Display number allocation utility
openhands/runtime/impl/*/runtime.py	Screen number configuration for all runtimes
openhands/runtime/utils/runtime_templates/singularity.j2	Updated base system setup with Xvfb dependencies
tests/nvidia/test_computer_use_tool.py	Demonstration test for computer use functionality
openhands/core/config/sandbox_config.py	Updated default base image to ubuntu:24.04

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

openhands/runtime/impl/singularity/singularity_runtime.py

Copilot · 2025-10-25T20:41:40Z

openhands/nvidia/reward.py


+        # Convert res to float score
        if isinstance(res, (int, float, bool)):
            score = float(res)


This changes the behavior from extracting the first element of a sequence (res[0]) to raising an error. If res can be a list/tuple in valid cases, this will break existing functionality. The change should either preserve the original behavior or be accompanied by evidence that res is never a sequence in valid scenarios.

Suggested change

score = float(res)

score = float(res)

elif isinstance(res, (list, tuple)) and len(res) > 0 and isinstance(res[0], (int, float, bool)):

score = float(res[0])

Copilot · 2025-10-25T20:41:40Z

openhands/runtime/action_execution_server.py

+    # import debugpy
+    # debugpy.listen(5678)
+    # logger.info('Waiting for debugger attach')
+    # debugpy.wait_for_client()
+    # logger.info('Debugger attached')
+    # debugpy.breakpoint()
+


Commented-out debugging code should be removed before merging to production. If this is needed for development, consider using environment variable-based conditional debugging or remove it entirely.

Suggested change

# import debugpy

# debugpy.listen(5678)

# logger.info('Waiting for debugger attach')

# debugpy.wait_for_client()

# logger.info('Debugger attached')

# debugpy.breakpoint()

Copilot · 2025-10-25T20:41:40Z

tests/nvidia/test_computer_use_tool.py

+
+    # Setup workspace (clean up old runs first)
+    workspace_dir = tempfile.mkdtemp(prefix='computer_use_demo_')
+    singularity_image_path = '/root/singularity_images/oh_v0.40.0_ubuntu_t_24.04.sif'


Hardcoded absolute path is not portable and will fail in different environments. Consider using a relative path, environment variable, or configuration parameter for the Singularity image location.

Suggested change

singularity_image_path = '/root/singularity_images/oh_v0.40.0_ubuntu_t_24.04.sif'

singularity_image_path = os.environ.get(

'SINGULARITY_IMAGE_PATH',

os.path.join(os.getcwd(), 'oh_v0.40.0_ubuntu_t_24.04.sif')

)

if not os.path.isfile(singularity_image_path):

print(f"❌ Singularity image not found at '{singularity_image_path}'. Please set the SINGULARITY_IMAGE_PATH environment variable or place the image in the current directory.")

return

Copilot · 2025-10-25T20:41:40Z

tests/nvidia/test_computer_use_tool.py

+        print(f'📸 Screenshots saved in: {screenshots_dir}')
+
+        # Save run GIF into logs directory
+        logs_dir = '/lustre/fsw/portfolios/llmservice/users/shaokunz/project/OpenHands_internal/logs'


Hardcoded user-specific absolute path should not be committed to the repository. This will fail in any other environment. Use a configurable path, environment variable, or temporary directory.

Suggested change

logs_dir = '/lustre/fsw/portfolios/llmservice/users/shaokunz/project/OpenHands_internal/logs'

logs_dir = os.environ.get('OPENHANDS_LOGS_DIR', tempfile.mkdtemp(prefix="openhands_logs_"))

Copilot · 2025-10-25T20:41:41Z

openhands/runtime/utils/runtime_templates/singularity.j2

+    # Regenerate lock file if pyproject.toml has changed
+    /openhands/micromamba/bin/micromamba run -n openhands poetry lock


Unconditionally regenerating the poetry lock file on every build will significantly increase build times. This should only run if pyproject.toml has actually changed, or this line should be removed if lock file regeneration is not necessary.

Suggested change

# Regenerate lock file if pyproject.toml has changed

/openhands/micromamba/bin/micromamba run -n openhands poetry lock

# Regenerate lock file only if pyproject.toml has changed or poetry.lock does not exist

if [ ! -f poetry.lock ] || [ pyproject.toml -nt poetry.lock ]; then

/openhands/micromamba/bin/micromamba run -n openhands poetry lock

fi

Copilot · 2025-10-25T20:41:41Z

openhands/runtime/action_execution_server.py

+
+                try:
+                    # Execute the line and capture any return value
+                    result = eval(line, {'__builtins__': {}}, namespace)


Using eval() on user input, even with a restricted namespace, is a security risk. While __builtins__ is emptied, there may still be ways to escape the sandbox through the provided namespace functions. Consider using ast.literal_eval() for safe evaluation or implementing a proper parser for the supported commands.

Copilot · 2025-10-25T20:41:41Z

openhands/nvidia/stem_agent/stem_handler.py

 )

-from openhands.nvidia.stem_agent.stem_utils import (
+from openhands.nvidia.stem_agent.stem_utils import (  # type: ignore[attr-defined]


The type: ignore[attr-defined] comment suggests there's a type checking issue that's being suppressed. This should be investigated and properly fixed rather than ignored, as it may indicate a real problem with the import or type annotations.

Suggested change

from openhands.nvidia.stem_agent.stem_utils import ( # type: ignore[attr-defined]

from openhands.nvidia.stem_agent.stem_utils import (

Signed-off-by: Yi Dong <doyend@gmail.com> Signed-off-by: skzhang1 <shaokunzhang529@gmail.com> Signed-off-by: Mingjie Liu <mingjiel@nvidia.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: skzhang1 <shaokunzhang529@gmail.com> Signed-off-by: Mingjie Liu <mingjiel@nvidia.com>

Signed-off-by: Mingjie Liu <mingjiel@nvidia.com>

jayl940712

Approve

yidong72 requested review from Copilot and skzhang1 October 25, 2025 20:40

Copilot AI reviewed Oct 25, 2025

View reviewed changes

skzhang1 force-pushed the osworld_support branch from e25d7be to f13aee7 Compare October 28, 2025 16:31

doyend and others added 7 commits October 28, 2025 13:12

fix problem

5a3fa6f

Signed-off-by: Yi Dong <doyend@gmail.com> Signed-off-by: skzhang1 <shaokunzhang529@gmail.com> Signed-off-by: Mingjie Liu <mingjiel@nvidia.com>

first computer use

6538f87

Signed-off-by: Yi Dong <doyend@gmail.com> Signed-off-by: skzhang1 <shaokunzhang529@gmail.com> Signed-off-by: Mingjie Liu <mingjiel@nvidia.com>

working

12c7010

Signed-off-by: Yi Dong <doyend@gmail.com> Signed-off-by: skzhang1 <shaokunzhang529@gmail.com> Signed-off-by: Mingjie Liu <mingjiel@nvidia.com>

add missing files

88eedad

Signed-off-by: Yi Dong <doyend@gmail.com> Signed-off-by: skzhang1 <shaokunzhang529@gmail.com> Signed-off-by: Mingjie Liu <mingjiel@nvidia.com>

Update openhands/runtime/impl/singularity/singularity_runtime.py

77d9cd2

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: skzhang1 <shaokunzhang529@gmail.com> Signed-off-by: Mingjie Liu <mingjiel@nvidia.com>

License & Readme & remove front & CICD etc.

eeb01cd

Signed-off-by: Mingjie Liu <mingjiel@nvidia.com>

License & Readme & remove front & CICD etc.

326bc17

jayl940712 force-pushed the osworld_support branch from 5be3927 to 326bc17 Compare October 28, 2025 18:17

Merge branch 'stable' into osworld_support

1ca5bd9

jayl940712 approved these changes Oct 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Osworld support #5

Osworld support #5

Uh oh!

yidong72 commented Oct 25, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Oct 25, 2025

Uh oh!

Copilot AI Oct 25, 2025

Uh oh!

Copilot AI Oct 25, 2025

Uh oh!

Copilot AI Oct 25, 2025

Uh oh!

Copilot AI Oct 25, 2025

Uh oh!

Copilot AI Oct 25, 2025

Uh oh!

Copilot AI Oct 25, 2025

Uh oh!

jayl940712 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

-    singularity_image_path = '/root/singularity_images/oh_v0.40.0_ubuntu_t_24.04.sif'
+    singularity_image_path = os.environ.get(
+        'SINGULARITY_IMAGE_PATH',
+        os.path.join(os.getcwd(), 'oh_v0.40.0_ubuntu_t_24.04.sif')
+    )
+    if not os.path.isfile(singularity_image_path):
+        print(f"❌ Singularity image not found at '{singularity_image_path}'. Please set the SINGULARITY_IMAGE_PATH environment variable or place the image in the current directory.")
+        return

	logs_dir = '/lustre/fsw/portfolios/llmservice/users/shaokunz/project/OpenHands_internal/logs'
	logs_dir = os.environ.get('OPENHANDS_LOGS_DIR', tempfile.mkdtemp(prefix="openhands_logs_"))

		# Regenerate lock file if pyproject.toml has changed
		/openhands/micromamba/bin/micromamba run -n openhands poetry lock

	from openhands.nvidia.stem_agent.stem_utils import ( # type: ignore[attr-defined]
	from openhands.nvidia.stem_agent.stem_utils import (

Osworld support #5

Are you sure you want to change the base?

Osworld support #5

Uh oh!

Conversation

yidong72 commented Oct 25, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

jayl940712 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants