Skip to content

[CORE] System Monitoring and Persistent Alert Management for Daemon #651

@sujay-d07

Description

@sujay-d07

Problem

Related to #427. Part 2/3 of the Features.

The cortexd daemon currently provides only basic IPC handlers (ping, version, config, shutdown) and lacks the ability to monitor system health or manage persistent alerts.

Without system monitoring capabilities, the daemon cannot:

  • Detect and report system health issues (high memory usage, disk space, failed services, etc.)
  • Track alerts persistently across daemon restarts
  • Provide proactive notifications about system problems
  • Enable users to acknowledge or dismiss alerts through the IPC interface

This limits the daemon's utility as a system management tool and prevents integration with monitoring workflows.

Proposed solution

Adds system monitoring and persistent alert management on top of the daemon.

System Monitoring

  • Implement a system monitoring service that tracks:
    • Memory usage and thresholds
    • Disk space utilization
    • Failed systemd services
    • System resource metrics
  • Run monitoring checks at configurable intervals
  • Generate alerts when thresholds are exceeded or issues are detected

Persistent Alert Management

  • Implement alert storage using SQLite database (similar to history.db pattern)
  • Store alerts in ~/.cortex/alerts.db or /var/lib/cortex/alerts.db
  • Implement IPC handlers for:
    • alerts - Retrieve all active alerts or alerts by filter
    • alerts.acknowledge - Mark an alert as acknowledged
    • alerts.dismiss - Permanently dismiss an alert
  • Persist alert state (active, acknowledged, dismissed) across daemon restarts
  • Support alert metadata (severity, timestamp, source, description)

Integration Points

  • Add monitoring service as a daemon service (following Service interface pattern)
  • Extend IPC handlers to support alert methods
  • Add configuration options for monitoring intervals and thresholds
  • Integrate with existing notification system (if applicable)

Example Usage

# Uses status IPC handler - Show daemon status
cortex daemon status

# Verbose status with health info - Detailed system health
cortex daemon status -v

# Uses health IPC handler - Check system health
cortex daemon health

# Uses alerts IPC handlers - List all alerts
cortex daemon alerts

# Filter by severity
cortex daemon alerts --severity <level>

# Uses alerts.acknowledge handler - Acknowledge all alerts
cortex daemon alerts --acknowledge-all

# Uses alerts.dismiss handler - Dismiss specific alert
cortex daemon alerts --dismiss <uuid>

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions