An automated system that picks up Jira issues, generates code fixes with Claude Code, opens pull requests on GitHub, and responds to reviewer feedback. Runs as Prow CI jobs on the HyperShift repository.
Pipeline Phases
Each agent run progresses through ordered phases: solve (write the fix), review (self-review the diff), fix (address review feedback), and pr-creation (open or update the PR). A single issue may have multiple sessions, each executing some or all phases.
Session
One Claude Code invocation within a pipeline phase. Each session has its own token usage, cost, duration, and tool call metrics. A single Jira issue can span multiple sessions across multiple phases.
Scraper
Background jobs that populate the dashboard database. Prow Scrape imports job runs, issues, and phase metrics from CI build logs. GitHub Sync refreshes PR state, diff stats, and review comments. Classification assigns severity and topic labels to review comments using Claude.
Issue Lifecycle
Issue
A Jira ticket the agent attempted to solve. Each issue is linked to one or more CI job runs and (if successful) a GitHub pull request.
PR State
The outcome of the agent's pull request: merged (accepted by reviewers), open (still under review), or closed (rejected or superseded without merging).
Merge Duration
Wall-clock time from PR creation to merge or close. Includes time waiting for reviewers, CI, and any fix cycles.
Review Cycles
The number of review-then-fix loops on a PR. Each time the agent runs a "fix" phase in response to reviewer feedback counts as one cycle. Lower is better — means the agent addresses feedback efficiently.
First-Pass Rate
Percentage of merged PRs that had zero "required change" review comments. A high first-pass rate means the agent is producing code that reviewers accept without requesting changes.
Quality & Cost
Quality Score
A 0-100 composite score for each issue, calculated from four components:
Comment severity (35 pts) — starts at 35, penalized by required_change (8pts each), question (4pts), suggestion (2pts), nitpick (1pt)
Review density (15 pts) — ratio of comments to lines changed; fewer comments per line = higher score
Topic penalties (10 pts) — starts at 10, penalized by security (5pts), logic_bug (5pts), test_gap (3pts), architecture_design (2pts), style (1pt)
Merge Rate
Percentage of agent-created PRs that were merged (accepted by reviewers) out of all PRs in the selected time range.
Total Cost
Claude API cost in USD summed across all pipeline phases for an issue. Includes input tokens, output tokens, and cache creation tokens.
Wasted Cost
Total cost spent on PRs that were closed without merging — money spent on work that wasn't accepted.
Avg Cost per Merged PR
Total cost of merged PRs divided by the number of merged PRs. Tracks the typical investment needed for a successful outcome.
Review Comments
Comment Classification
Each human review comment is classified by an LLM into a severity and topic. Classifications can be manually overridden by dashboard users.
Severity
How critical a review comment is:
required_change — must be fixed before merge
suggestion — recommended improvement, not blocking
question — reviewer asking for clarification
nitpick — minor style or formatting preference
Topic
What area a review comment addresses:
security — security vulnerabilities or concerns
logic_bug — incorrect behavior or logic errors
test_gap — missing or insufficient test coverage
architecture_design — structural or design issues
error_handling — missing or incorrect error handling
naming_readability — unclear names or hard-to-read code
style — code formatting or convention preferences
documentation — missing or incorrect docs/comments
performance — efficiency or resource usage concerns
Topic Merge Rate
For each comment topic, the percentage of comments with that topic that appeared on PRs which ultimately merged. Low merge rate for a topic means that type of feedback correlates with PR rejection.
Complexity
Lines Added / Deleted
Number of lines added or removed in the PR diff, as reported by GitHub.
Files Changed
Number of files modified in the PR.
Complexity Delta
Change in cyclomatic complexity introduced by the PR, measured with gocyclo. Positive means the code got more complex; negative means it was simplified.
Telemetry (Internals)
Input / Output Tokens
Number of tokens sent to (input) and received from (output) the Claude model in a session.
Cache Hit Rate
Percentage of input tokens served from Claude's prompt cache rather than reprocessed. Higher rates reduce cost and latency.
Tool Calls
Number of tool invocations (Read, Edit, Bash, Grep, etc.) the agent made during a session.
Turns
Number of conversation turns (model response cycles) in a session.
Subagents
Number of sub-agent processes spawned by the main agent during a session (e.g., for parallel research or code review).
TTFT (Time to First Token)
Latency from sending a request to receiving the first token back from the model. Affected by prompt size and cache hit rate.
Phase Success Rate
Percentage of sessions in a pipeline phase that completed without error. Shows which phase is most likely to fail.