Опишете задачата
Към всички skills

evaluate-presets

mikeyobrien · Тестване

Use when testing Ralph's hat collection presets, validating preset configurations, or auditing the preset library for bugs and UX issues.

Оригинален текст · няма потвърден превод на български
Платформи: Не са посочениНиво: Не е посоченоВ индекса: 22.09.2026 г.Обновен в GitHub: 10.09.2026 г.

Теми: ai · ai-agents · ai-agents-framework · ai-developer-tools · claude-code · claude-code-cli · codex-cli · development-tools · development-workflow · gemini-cli · kiro · kiro-cli · opencode · ralph-loop · ralph-wiggum

Първо прочетете инструкцията.

Файлът се зарежда от посочения източник само по ваше действие. Можете да го прочетете тук, да го копирате или да го запазите като .md.

Оригинален SKILL.md
GitHub хранилище
Звезди на репото
3155
Лиценз
MIT
Последна проверка на записа
22.09.2026 г.

Добавете в своя инструмент.

Командата отваря избора на skills от репозитория. Прочетете файла и условията на автора преди изпълнение.

npx skills add mikeyobrien/ralph-orchestrator

Страницата копира текст. Не инсталира и не изпълнява нищо на устройството ви.

Откъс от SKILL.md

Началото на файла на автора, без превод.

Evaluate Presets

Overview

Systematically test all hat collection presets using shell scripts. Direct CLI invocation—no meta-orchestration complexity.

When to Use

  • Testing preset configurations after changes
  • Auditing the preset library for quality
  • Validating new presets work correctly
  • After modifying hat routing logic

Quick Start

Evaluate a single preset:

./tools/evaluate-preset.sh tdd-red-green claude

Evaluate all presets:

./tools/evaluate-all-presets.sh claude

Arguments:

  • First arg: preset name (without .yml extension)
  • Second arg: backend (claude or kiro, defaults to claude)

Bash Tool Configuration

IMPORTANT: When invoking these scripts via the Bash tool, use these settings:

  • Single preset evaluation: Use timeout: 600000 (10 minutes max) and run_in_background: true
  • All presets evaluation: Use timeout: 600000 (10 minutes max) and run_in_background: true

Since preset evaluations can run for hours (especially the full suite), always run in background mode and use the TaskOutput tool to check progress periodically.

Example invocation pattern:

Bash tool with:
  command: "./tools/evaluate-preset.sh tdd-red-green claude"
  timeout: 600000
  run_in_background: true

After launching, use TaskOutput with block: false to check status without waiting for completion.

What the Scripts Do

evaluate-preset.sh

  1. Loads test task from tools/preset-test-tasks.yml (if yq available)
  2. Creates merged config with evaluation settings
  3. Runs Ralph with --record-session for metrics capture
  4. Captures output logs, exit codes, and timing
  5. Extracts metrics: iterations, hats activated, events published

Output structure:

.eval/
├── logs/<preset>/<timestamp>/
│   ├── output.log          # Full stdout/stderr
│   ├── session.jsonl       # Recorded session
│   ├── metrics.json        # Extracted metrics
│   ├── environment.json    # Runtime environment
│   └── merged-config.yml   # Config used
└── logs/<preset>/latest -> <timestamp>

evaluate-all-presets.sh

Runs all 12 presets sequentially and generates a summary:

.eval/results/<suite-id>/
├── SUMMARY.md              # Markdown report
├── <preset>.json           # Per-preset metrics
└── latest -> <suite-id>

Presets Under Evaluation

Целият SKILL.md в GitHub