Danny/kernel 742 create yutori n1 computer use cli templates (ts/python) #89

dprevoznik · 2026-01-21T22:58:01Z

Add Yutori n1 Computer Use CLI Templates

This PR adds new CLI templates for Yutori's n1 computer use model, enabling users to quickly scaffold browser automation projects using Kernel's infrastructure.

New Templates

TypeScript: kernel create --template ts-yutori-cua
Python: kernel create --template python-yutori-cua

Features

Both templates include:

Agentic sampling loop with n1's OpenAI-compatible API
Computer tool mapping n1 actions (click, type, scroll, drag, hover, key_press, wait, refresh, go_back, goto_url, stop) to Kernel's Computer Controls API
Coordinate scaling from n1's 1000×1000 relative space to actual viewport dimensions
Session management with replay recording support

Dual Screenshot Modes

Mode	Description
`computer_use` (default)	Uses Kernel's Computer Controls screenshot API (stable)
`playwright`	Uses CDP WebSocket connection for viewport-only screenshots without browser chrome, optimized for n1's training data per Yutori's documentation

Implementation Details

Model: n1-preview-2025-11 outputs coordinates in 1000×1000 space
Viewport: 1200×800 at 25Hz (closest to Yutori's recommended 1280×800)

With Playwright Mode for viewport-only screenshots

kernel invoke ts-yutori-cua cua-task --payload '{"query": "...", "mode": "playwright"}'

Files Changed

pkg/templates/typescript/yutori-computer-use/ - TypeScript template
pkg/templates/python/yutori-computer-use/ - Python template
pkg/create/templates.go - Template registration

Closes KERNEL-742

Note

Introduces Yutori n1 templates to quickly scaffold browser-automation apps using Kernel and Yutori’s OpenAI-compatible API.

New templates: typescript/yutori-computer-use and python/yutori-computer-use with index.ts/main.py, sampling loops, computer tools, Playwright tools, and session managers (replay support)
Dual modes: computer_use (full VM screenshots) and playwright (viewport-only via CDP) with coordinate scaling from 1000×1000 to viewport
CLI integration: Registers yutori-computer-use in pkg/create/templates.go, adds invoke/deploy commands, ordering, and env requirements (YUTORI_API_KEY)
QA updates: Expands matrix, create/deploy/invoke commands, automated tests (adds Yutori cases), and ignores qa-* in .gitignore

^{Written by Cursor Bugbot for commit a0ae834. This will update automatically on new commits. Configure here.}

Add new CLI templates for Yutori's n1 computer use model, enabling users to quickly scaffold browser automation projects using Kernel's infrastructure. Templates (TypeScript & Python): - Agentic sampling loop with n1's OpenAI-compatible API - Computer tool mapping n1 actions (click, type, scroll, drag, etc.) to Kernel's Computer Controls API - Coordinate scaling from n1's 1000x1000 relative space to actual viewport - Session management with replay recording support - read_texts_and_links action using Playwright execution API (with fallback) Key implementation details: - n1 requires screenshots sent with role 'observation' (not 'user') - Model: n1-preview-2025-11 outputs coordinates in 1000x1000 space - Viewport: 1200x800 at 25Hz (closest to Yutori's recommended 1280x800) - Navigation actions (refresh, go_back, goto_url) use keyboard shortcuts via Computer Controls since n1 doesn't use Playwright directly Also updated: - .gitignore: Added qa-* to exclude QA testing directories - pkg/create/templates.go: Registered new yutori-computer-use templates - .cursor/commands/qa.md: Added Yutori templates to QA testing matrix Closes KERNEL-742

Replace page.accessibility.snapshot() with page._snapshotForAI() which is specifically designed for AI agents and documented in Kernel's MCP server. The previous implementation used the experimental/deprecated accessibility API which failed silently and fell back to screenshot-only mode. _snapshotForAI() returns a structured representation of the page optimized for LLM consumption, including visible text, interactive elements (links, buttons, inputs), and page structure - exactly what n1 needs for reading texts and saving URLs for citation.

Add PlaywrightComputerTool adapter that connects via CDP WebSocket for browser-only screenshots, optimized for Yutori n1's training data per their documentation recommendations. Changes: - Add PlaywrightComputerTool class (TS + Python) using CDP connection - Add 'mode' parameter to sampling loop ('computer_use' | 'playwright') - Default to 'computer_use' mode (stable); 'playwright' is opt-in - Add configurable viewport dimensions (1200x800) - Expose cdp_ws_url from session for Playwright connection - Add playwright-core (TS) and playwright (Python) dependencies The playwright mode provides viewport-only screenshots without OS UI or browser chrome, improving n1 model performance per Yutori's docs: https://docs.yutori.com/reference/n1#screenshot-requirements

Add templates + modes for Yutori to QA file

Fix drag operations that previously weren't working properly on Playwright mode operations.

Use ariaSnapshot instead of the existing method, as ariaSnapshot is stably available in both Python and TypeScript versions.

Issue: The ComputerTool.screenshot() method was a synchronous function, but: The N1ComputerToolProtocol expected it to be async The PlaywrightComputerTool.screenshot() was async The loop.py code tried to await it Fix: Changed def screenshot() to async def screenshot() Updated all handler methods to await self.screenshot() instead of return self.screenshot()

Update default delays for actions and screenshots

… moving. Clarified instructions for both computer_use and playwright modes to enhance user understanding and execution accuracy.

The cleanup removed ~300 lines of redundant inline comments and verbose method docstrings while keeping the useful class-level documentation you restored. The templates now match the minimal-comment style of the existing anthropic/openai templates in the codebase.

@masnwilliams

#88) This PR updates the Go SDK to cee2050be3f8136505d41c20c2903dfca2cbc479 and adds CLI commands for new SDK methods. ## SDK Update - Updated kernel-go-sdk to cee2050be3f8136505d41c20c2903dfca2cbc479 ## Coverage Analysis This PR was generated by performing a full enumeration of SDK methods and CLI commands. ## New Commands - `kernel credential-providers list` - List configured external credential providers - `kernel credential-providers get <id>` - Get a credential provider by ID - `kernel credential-providers create` - Create a new credential provider (supports 1Password) - `kernel credential-providers update <id>` - Update a credential provider's configuration - `kernel credential-providers delete <id>` - Delete a credential provider - `kernel credential-providers test <id>` - Test a credential provider connection ## Breaking Changes Fixed - Fixed `browsers.Get()` calls to pass new required `BrowserGetParams` parameter Triggered by: kernel/kernel-go-sdk@cee2050 Reviewer: @masnwilliams  --- > [!NOTE] > Introduces new CLI surfaces and updates for latest SDK. > > - **Agent Auth CLI**: `kernel agents auth` with `create/get/list/delete`, `invocations {create/get/exchange/submit}`, and end‑to‑end `run` flow (auto field submission, TOTP, optional live view); docs and examples added to `README.md`. > - **Credential Providers CLI**: `kernel credential-providers {list/get/create/update/delete/test}` (supports 1Password), wired into root. > - **Browsers API updates**: adapt to SDK breaking change (`browsers.Get` now requires `BrowserGetParams`); add `process resize` and filesystem watch (`fs watch start/stop/events`) commands; tests updated accordingly. > - **Dependencies**: bump `kernel-go-sdk` to cee2050… and add `pquerna/otp`; regenerate `go.sum`. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 0b27df6. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>  --------- Co-authored-by: Mason Williams <43387599+masnwilliams@users.noreply.github.com> Co-authored-by: Cursor Agent <cursor-agent@kernel.sh> Co-authored-by: Cursor Agent <cursor-agent@onkernel.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>

…se-cli-templates-typescript

dprevoznik · 2026-01-21T23:17:27Z

Working on fixing comments from bugbot then will request review

… and remove unused dependencies from Python and TypeScript templates.

…se-cli-templates-typescript

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

pkg/templates/typescript/yutori-computer-use/tools/playwright-computer.ts

pkg/templates/python/yutori-computer-use/tools/playwright_computer.py

…se-cli-templates-typescript

dprevoznik · 2026-01-23T22:39:05Z

@Sayan- no rush, but could use a quick review on these templates when possible 🙏

Running /qa is a pretty solid way to try out the different templates to ensure they deploy and run the actions (I usually have it do the magnitasks.com example which uses a bunch of features.

Sayan-

great stuff!

Sayan- · 2026-01-23T23:47:24Z

pkg/templates/typescript/yutori-computer-use/session.ts

+  recordReplay?: boolean;
+  /** Grace period in seconds before stopping replay */
+  replayGracePeriod?: number;
+  /** Viewport width (default: 1280 per Yutori recommendation) */


nit: comment says "default: 1280" but the actual default on line 40 is 1200. consider updating to match the other comments (e.g., "default: 1200, closest to Yutoris 1280 recommendation").

dprevoznik and others added 12 commits January 19, 2026 21:34

Update qa.md

748aa2b

Add templates + modes for Yutori to QA file

Fix Drag Operation on Playwright Mode

8e2df1b

Fix drag operations that previously weren't working properly on Playwright mode operations.

Fix read_text_and_links action

71e7a85

Use ariaSnapshot instead of the existing method, as ariaSnapshot is stably available in both Python and TypeScript versions.

Update default delays

1ae6ddb

Update default delays for actions and screenshots

Update Yutori CUA tasks in qa.md to specify dragging items instead of…

0a88d32

… moving. Clarified instructions for both computer_use and playwright modes to enhance user understanding and execution accuracy.

ran deslop

8cc1683

The cleanup removed ~300 lines of redundant inline comments and verbose method docstrings while keeping the useful class-level documentation you restored. The templates now match the minimal-comment style of the existing anthropic/openai templates in the codebase.

Merge branch 'main' into danny/kernel-742-create-yutori-n1-computer-u…

0082c56

…se-cli-templates-typescript

dprevoznik marked this pull request as ready for review January 21, 2026 23:01

This comment was marked as outdated.

Sign in to view

dprevoznik added 2 commits January 21, 2026 20:09

Handle cursor bugbot comments. Update viewport width in sampling loop…

1277f3e

… and remove unused dependencies from Python and TypeScript templates.

Merge branch 'main' into danny/kernel-742-create-yutori-n1-computer-u…

7e4ce52

…se-cli-templates-typescript

cursor bot reviewed Jan 22, 2026

View reviewed changes

pkg/templates/typescript/yutori-computer-use/tools/playwright-computer.ts Show resolved Hide resolved

pkg/templates/python/yutori-computer-use/tools/playwright_computer.py Show resolved Hide resolved

Merge branch 'main' into danny/kernel-742-create-yutori-n1-computer-u…

a0ae834

…se-cli-templates-typescript

dprevoznik requested a review from Sayan- January 23, 2026 22:37

Sayan- approved these changes Jan 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Danny/kernel 742 create yutori n1 computer use cli templates (ts/python) #89

Danny/kernel 742 create yutori n1 computer use cli templates (ts/python) #89

Uh oh!

dprevoznik commented Jan 21, 2026 •

edited by cursor bot

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

dprevoznik commented Jan 21, 2026

Uh oh!

cursor bot left a comment

Uh oh!

Uh oh!

Uh oh!

dprevoznik commented Jan 23, 2026

Uh oh!

Sayan- left a comment

Uh oh!

Sayan- Jan 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Danny/kernel 742 create yutori n1 computer use cli templates (ts/python) #89

Are you sure you want to change the base?

Danny/kernel 742 create yutori n1 computer use cli templates (ts/python) #89

Uh oh!

Conversation

dprevoznik commented Jan 21, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add Yutori n1 Computer Use CLI Templates

New Templates

Features

Dual Screenshot Modes

Implementation Details

With Playwright Mode for viewport-only screenshots

Files Changed

Uh oh!

This comment was marked as outdated.

Uh oh!

dprevoznik commented Jan 21, 2026

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dprevoznik commented Jan 23, 2026

Uh oh!

Sayan- left a comment

Choose a reason for hiding this comment

Uh oh!

Sayan- Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dprevoznik commented Jan 21, 2026 •

edited by cursor bot

Loading