Skip to content

Conversation

@basnijholt
Copy link
Owner

Summary

  • Adds a new council subcommand inspired by Karpathy's llm-council
  • Enables collaborative AI deliberation through a 3-stage process:
    1. Stage 1: Query multiple models in parallel
    2. Stage 2: Models peer-review anonymized responses using structured output (JSON mode)
    3. Stage 3: Chairman model synthesizes a final answer

Features

  • Works with any OpenAI-compatible API (OpenRouter, Ollama, etc.)
  • Structured output for reliable ranking extraction
  • Rich console output showing full deliberation process
  • --no-ranking flag to skip peer review for faster results
  • --final-only flag to show only the synthesis
  • Auto-copy final answer to clipboard

Usage

# Basic usage with OpenRouter
agent-cli council "What is the best way to learn programming?" \
    --openai-base-url https://openrouter.ai/api/v1 \
    --openai-api-key sk-or-...

# Custom models
agent-cli council "..." --models "gpt-4o,claude-3-opus,gemini-1.5-pro"

# Skip ranking for faster results
agent-cli council "..." --no-ranking

# Output as JSON
agent-cli council "..." --json

Test plan

  • Test with OpenRouter using multiple models
  • Test --no-ranking flag skips stage 2
  • Test --final-only shows only synthesis
  • Test --json output format
  • Test clipboard auto-copy

Inspired by Karpathy's llm-council project, this adds a new `council`
subcommand that enables collaborative AI deliberation:

- Stage 1: Query multiple models in parallel
- Stage 2: Models peer-review anonymized responses with structured output
- Stage 3: Chairman model synthesizes final answer

Features:
- Works with any OpenAI-compatible API (OpenRouter, Ollama, etc.)
- Structured output (JSON mode) for reliable ranking extraction
- Rich console output showing full deliberation process
- --no-ranking flag to skip peer review for faster results
- --final-only flag to show only the synthesis
- Auto-copy final answer to clipboard
- Renamed run_council -> _run_council
- Renamed render_council_result -> _render_council_result
- Added comprehensive test suite with 18 tests covering:
  - Data model creation
  - Ranking aggregation logic
  - Formatting functions
  - CLI command invocation
  - Integration tests with mocked API
Use NO_COLOR and TERM=dumb env vars in CliRunner to prevent
Rich console from emitting ANSI escape codes in test output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants