Skip to content

NO-JIRA: Claude tool - etcd troubleshooting skill #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
fonta-rh wants to merge 19 commits into
base: main
Choose a base branch
from
Open

NO-JIRA: Claude tool - etcd troubleshooting skill #32

fonta-rh wants to merge 19 commits into from

Conversation

@fonta-rh
Copy link
Contributor

@fonta-rh fonta-rh commented Oct 29, 2025

Add etcd troubleshooting skill for Claude Code

Adds a comprehensive Claude Code skill that helps troubleshoot etcd issues on two-node fencing clusters.

The skill enables automated diagnosis and remediation of common etcd/Pacemaker problems.

New feature: Claude Code Skill (.claude/commands/etcd/):

  • Interactive troubleshooting capability with systematic decision trees
  • Automated diagnostic data collection from Pacemaker, etcd, and OpenShift
  • Analysis guidelines for cluster state, resource failures, and error patterns
  • Remediation recommendations with verification steps
  • Permission framework for safe read-only operations vs. requiring approval for state changes

Diagnostic Tools:

  • Ansible playbooks for validation and data collection
  • Shell scripts with automatic proxy.env detection for cluster access
  • Master orchestration script collecting both VM and cluster-level diagnostics

Helper:

  • force-new-cluster.yml - Automated recovery for cluster ID mismatches and split-brain scenarios

Documentation:

  • Etcd operations guide (clustering, recovery, monitoring, failures)
  • Pacemaker administration reference
  • Comprehensive skill usage and permission configuration docs

Tested with:

  • Cluster ID mismatch recovery
  • Resource failure cleanup
  • Failed learner rejoins
  • Transient CIB communication errors

@openshift-ci openshift-ci bot requested review from clobrano and eggfoobar October 29, 2025 10:27
@openshift-ci
Copy link

openshift-ci bot commented Oct 29, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fonta-rh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 29, 2025
@fonta-rh fonta-rh changed the title Claude tool: etcd troubleshooting skill NO-JIRA: Claude tool - etcd troubleshooting skill Oct 29, 2025
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 29, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Oct 29, 2025

@fonta-rh: This pull request explicitly references no jira issue.

Details

In response to this:

Add etcd troubleshooting skill for Claude Code

Adds a comprehensive Claude Code skill that helps troubleshoot etcd issues on two-node fencing clusters.

The skill enables automated diagnosis and remediation of common etcd/Pacemaker problems.

New feature: Claude Code Skill (.claude/commands/etcd/):

  • Interactive troubleshooting capability with systematic decision trees
  • Automated diagnostic data collection from Pacemaker, etcd, and OpenShift
  • Analysis guidelines for cluster state, resource failures, and error patterns
  • Remediation recommendations with verification steps
  • Permission framework for safe read-only operations vs. requiring approval for state changes

Diagnostic Tools:

  • Ansible playbooks for validation and data collection
  • Shell scripts with automatic proxy.env detection for cluster access
  • Master orchestration script collecting both VM and cluster-level diagnostics

Helper:

  • force-new-cluster.yml - Automated recovery for cluster ID mismatches and split-brain scenarios

Documentation:

  • Etcd operations guide (clustering, recovery, monitoring, failures)
  • Pacemaker administration reference
  • Comprehensive skill usage and permission configuration docs

Tested with:

  • Cluster ID mismatch recovery
  • Resource failure cleanup
  • Failed learner rejoins
  • Transient CIB communication errors

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@fonta-rh fonta-rh added tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. and removed jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels Oct 29, 2025
clobrano
clobrano requested changes Nov 28, 2025
View reviewed changes
Copy link
Contributor

@clobrano clobrano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some comments

.claude/commands/etcd/README.md

```bash
# From repository root
./.claude/commands/etcd/scripts/collect-all-diagnostics.sh
Copy link
Contributor

@clobrano clobrano Nov 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having scripts in an hidden directories works against discoverability. What's the reasoning behind this choice?
Also, .claude/commands is a special one for Claude. One could try to execute it as a "custom command":

[in Claude CLI] > /etcd:scripts:collect-all-diagnostic.sh

I'm curious to see if this works :D

.claude/commands/etcd/pacemaker/podman-etcd.txt
@@ -0,0 +1,2052 @@
#!/bin/sh
Copy link
Contributor

@clobrano clobrano Nov 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to keep this file in sync automatically?

.claude/commands/etcd/playbooks/collect-diagnostics.yml
when: podman_inspect.rc == 0

- name: Get podman logs for etcd
ansible.builtin.command: podman logs --tail 100 etcd
Copy link
Contributor

@clobrano clobrano Nov 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why limit to 100? Would it be too much to get all logs?

release-notes.md
Comment on lines +9 to +10
- Added `force-new-cluster.yml` for automated etcd cluster recovery via CIB attributes
- Ansible conversion of Carlo Lobrano's shell script using `cluster_vms` inventory group
Copy link
Contributor

@clobrano clobrano Nov 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a warning about the playbook assuming the first node in the inventory is the leader.
Another option would be to add some confirmation step in the playbook itself, to ensure the target node is correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@clobrano clobrano clobrano requested changes

@eggfoobar eggfoobar Awaiting requested review from eggfoobar

Assignees

@clobrano clobrano

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@fonta-rh @openshift-ci-robot @clobrano