feat: refactor rules generation to ai-context with skill output #1199

stefanjudis · 2026-01-29T17:29:46Z

Affected Components

Notes for the Reviewer

This PR changes how Checkly AI context files (aka rules.md) are generated to align with upcoming skill conventions in the AI ecosystem. The primary concern is to provide a SKILL.md for MaC and be compatible with Vercel's new skills tooling:

npx skills add checkly/checkly-cli
npx skills add checkly/checkly-cli --skill monitoring

For that to work, the repo needs to include a skills directory in the root, which contains all the provided markdown skills (skills/monitoring/SKILL.md). For that, I repurposed the script of the checkly.rules.md and reached out of the cli repo to write files outside of /packages/cli. It's not pretty, and I'm open to ideas for doing it differently.

Generally, I changed the naming convention from rules to ai-context because the script now generates our "rules" (rules.md) and a Checkly "monitoring" skill (skills/monitoring/SKILL.md). The content is more or less the same, though. I've chosen the monitoring skill name to be flexible in creating more skills (e.g. a playwright-best-practices skill) in the future. We'll see how this plays out.

Additionally, I extracted the configuration code from the actual script into its own file so that there's a single config file.

Content-wise, I expect multiple iterations on the new "context" files, but I still made some adjustments on the way:

checkly.config code is now part of the generated context files. To do so, the code config now supports inline code samples bc checkly.config and pwCheckSuites aren't part of the exported files used for the existing code generation.
The manual links to construct docs were replaced with .md routes defined in the context config.

Resolves Slack conversation and MAR-504

@sorccu

The generated skills/monitoring/SKILL.md is not included in this PR (yet), it needs to be generated and committed on CLI release to be available for tools like npx skills add. How do you think we should go about it? Commit in the prebuild hook, quick'n'dirty? 🤔

To check later

We need to make sure that the configuration in the marketing site still works and provides the generated file from GitHub.

"source": "/docs/ai/checkly.rules.md",
"destination": "https://github.com/checkly/checkly-cli/releases/latest/download/checkly.rules.md"

Rename src/rules to src/ai-context and enhance the context generation: - Extract example configs to separate context.ts module with doc references - Generate both checkly.rules.md and skills/monitoring/SKILL.md - Add YAML frontmatter for skill metadata (stripped from rules output) - Support inline exampleConfig in addition to exampleConfigPath

Copilot

Pull request overview

Refactors how Checkly AI context (“rules.md”) content is generated so it can also produce a Vercel-compatible SKILL.md under a root-level skills/ directory.

Changes:

Move rules/context sources to src/ai-context/ and update the rules command to read from dist/ai-context/checkly.rules.md.
Replace the old compile-rules.ts pipeline with compile-ai-context.ts, generating both dist/ai-context/checkly.rules.md (without frontmatter) and skills/monitoring/SKILL.md (with frontmatter).
Add a centralized EXAMPLE_CONFIGS configuration and new fixtures for the generation flow.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/cli/src/commands/rules.ts	Updates base rules path to new `dist/ai-context` output location.
packages/cli/src/ai-context/context.ts	Adds centralized example/placeholder config (including inline code samples + doc references).
packages/cli/src/ai-context/context.fixtures.json	Adds fixtures used to generate example code via `import plan`.
packages/cli/src/ai-context/checkly.context.template.md	Introduces skill-style frontmatter and adjusts template to rely on injected examples.
packages/cli/scripts/compile-rules.ts	Removes the old rules compiler script.
packages/cli/scripts/compile-ai-context.ts	Adds new compiler that outputs both `SKILL.md` and stripped `checkly.rules.md`.
packages/cli/package.json	Renames generation script and wires it into `prepare`.

Comments suppressed due to low confidence (3)

packages/cli/src/ai-context/checkly.context.template.md:108

There’s still a hard-coded “Reference [the docs…]” link here, while other sections moved references into the generated example blocks. For consistency (and to avoid duplicated/competing references), consider removing this link and relying on the per-example reference injected by the compiler.
packages/cli/src/ai-context/checkly.context.template.md:132
Grammar: “All alert are described…” should be “All alerts are described…”. Since this template content is distributed to users/AI tooling, it’s worth fixing the wording.
packages/cli/src/ai-context/checkly.context.template.md:79
This line reads awkwardly/incorrectly (“if your tasked”). Please fix the grammar (e.g., “Use pwProjects if you’re tasked with reusing a Playwright project.”) since this text is shipped in generated AI context.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-29T17:58:51Z

packages/cli/src/ai-context/context.ts

+      'resources/browser-checks/example-browser-check/example-browser-check.check.ts',
+    reference: 'https://www.checklyhq.com/docs/constructs/browser-check.md',
+  },
+  PLAYRIGHT_CHECK: {


The config key is spelled PLAYRIGHT_CHECK but everything else refers to “Playwright”. Renaming this to PLAYWRIGHT_CHECK would avoid confusion and prevent future lookups by key from breaking due to the typo.

Suggested change

PLAYRIGHT_CHECK: {

PLAYWRIGHT_CHECK: {

Copilot · 2026-01-29T17:58:52Z

packages/cli/src/ai-context/context.ts

+  name: "Multi-browser check suite",
+  playwrightConfigPath: "./playwright.config.ts",
+  // Playwright Check Suites support all browsers
+  // defined in your \`playwight.config\`


In the embedded Playwright example, the comment references playwight.config (typo). This should be playwright.config to avoid propagating incorrect file names into generated code.

Suggested change

// defined in your \`playwight.config\`

// defined in your \`playwright.config\`

Copilot · 2026-01-29T17:58:52Z

packages/cli/scripts/compile-ai-context.ts

+const RULES_OUTPUT_DIR = join(__dirname, '../dist/ai-context')
+const SKILL_OUTPUT_DIR = join(__dirname, '../../../skills/monitoring')
+


SKILL_OUTPUT_DIR points outside the packages/cli workspace (../../../skills/monitoring). Because this script runs from prepare, builds in environments that only have the package folder (or read-only parent dirs) can fail or leave unexpected artifacts. Consider making the skill output optional (env flag) or moving skill generation to a dedicated release step so normal builds only write under dist/.

@sorccu This is similar to my concerns, I phrased in the description. I'm up for ideas. :)

We could of course, also write to the dist folder in /cli and then "snatch" it from the monorepo config, but yeah... Very keen on another opinion.

Copilot · 2026-01-29T17:58:52Z

packages/cli/scripts/compile-ai-context.ts

+const SKILL_OUTPUT_DIR = join(__dirname, '../../../skills/monitoring')
+
+function stripYamlFrontmatter (content: string): string {
+  const frontmatterRegex = /^---\n[\s\S]*?\n---\n+/


stripYamlFrontmatter only matches LF (\n). If the template ever uses CRLF (\r\n) or has a UTF-8 BOM, the YAML frontmatter won’t be stripped and the generated checkly.rules.md will include it. Consider making the regex tolerant of \r?\n (and optionally trimming a BOM) to keep output stable across platforms.

Suggested change

const frontmatterRegex = /^---\n[\s\S]*?\n---\n+/

const frontmatterRegex = /^(?:\uFEFF)?---\r?\n[\s\S]*?\r?\n---\r?\n+/

stefanjudis added 4 commits January 29, 2026 18:29

chore: remove ABOUTME comments from ai-context files

8b87896

fix: add missing playwright check template placeholder

1696f53

fix: add missing imports to inline example configs

9addc7a

stefanjudis marked this pull request as ready for review January 29, 2026 17:53

stefanjudis requested a review from Copilot January 29, 2026 17:53

Copilot started reviewing on behalf of stefanjudis January 29, 2026 17:53 View session

stefanjudis requested review from sorccu and removed request for sorccu January 29, 2026 17:54

Copilot AI reviewed Jan 29, 2026

View reviewed changes

stefanjudis added 2 commits January 29, 2026 19:03

fix: update CI artifact path from rules to ai-context

ad7dc5f

fix: correct typos in PLAYWRIGHT_CHECK config

160e513

stefanjudis requested a review from sorccu January 29, 2026 18:07

fix: handle CRLF line endings in YAML frontmatter stripping

5b61b07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: refactor rules generation to ai-context with skill output #1199

feat: refactor rules generation to ai-context with skill output #1199

Uh oh!

stefanjudis commented Jan 29, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 29, 2026

Uh oh!

Copilot AI Jan 29, 2026

Uh oh!

Copilot AI Jan 29, 2026

Uh oh!

stefanjudis Jan 29, 2026 •

edited

Loading

Uh oh!

Copilot AI Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	// defined in your \`playwight.config\`
	// defined in your \`playwright.config\`

		const RULES_OUTPUT_DIR = join(__dirname, '../dist/ai-context')
		const SKILL_OUTPUT_DIR = join(__dirname, '../../../skills/monitoring')

	const frontmatterRegex = /^---\n[\s\S]*?\n---\n+/
	const frontmatterRegex = /^(?:\uFEFF)?---\r?\n[\s\S]*?\r?\n---\r?\n+/

feat: refactor rules generation to ai-context with skill output #1199

Are you sure you want to change the base?

feat: refactor rules generation to ai-context with skill output #1199

Uh oh!

Conversation

stefanjudis commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Affected Components

Notes for the Reviewer

To check later

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

stefanjudis Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stefanjudis commented Jan 29, 2026 •

edited

Loading

stefanjudis Jan 29, 2026 •

edited

Loading