Designing a Multi-Agent Pipeline: Log Analyzer to Diagnosis to Remediation

PipelineHealer is a pipeline with constraints, not one prompt over a log dump.

Early experiments sent raw GitHub Actions output to a model and asked for a fix. Results mixed plausible guesses, half-right fixes, and confident hallucinations.

The fix: narrow agents passing typed data between steps. Common failure modes should resolve without a model; the model handles the rest.

The Pipeline

The core flow is a simple chain:

1. Log Analyzer: fetches CI logs and extracts structured signals (error lines, warning lines, key events). Does not diagnose.
2. Diagnosis: maps those signals into a failure category with confidence. Does not fix.
3. Remediation: generates a plan and either opens a PR or creates an issue. Does not guess from raw logs.
4. Orchestrator: coordinates the steps, persists state, and enforces guardrails.

This chain is documented in the PipelineHealer architecture diagram, but the important part is the interface between agents: everything is a Pydantic model, not free-form text.

The Rules That Prevent Hallucinations

These are the constraints that made the system behave:

• The Log Analyzer only extracts. If it starts diagnosing, you lose debuggability immediately.
• The Diagnosis agent sees a summary (error lines + job names + key events), not full raw logs.
• The Remediation agent is gated on diagnosis confidence. Low confidence means "create an issue", not "ship a PR".
• Pattern-based detection runs first for obvious cases; the model is a fallback for nuance.
• The orchestrator defaults to creating an issue when uncertain.

Default outcomes matter more than agent count.

Typed contracts

Data shape between steps:

# Simplified view
class LogAnalysis(BaseModel):
    job_name: str
    error_lines: list[str]
    warning_lines: list[str]
    key_events: list[str]
    summary: str

class Diagnosis(BaseModel):
    failure_type: FailureType
    root_cause: str
    confidence: float
    is_auto_fixable: bool
    affected_files: list[str]
    suggested_fix: str
    error_details: dict[str, Any]

class RemediationPlan(BaseModel):
    action: RemediationAction  # create_pr | create_issue | retry_workflow
    description: str
    file_changes: list[dict[str, Any]] | None
    issue_title: str | None
    issue_body: str | None

Typed contracts give you two advantages:

• You can write unit tests for the pipeline without calling the model.
• You can evolve the system without accidentally changing the meaning of a field.

Remediation Actions (What the System Is Allowed to Do)

At this early stage, PipelineHealer's action policy covered five core failure types:

• Dependency issues: PR (auto-fix)
• Lint/format issues: PR (auto-fix)
• Test failures: issue
• Build config errors: issue
• Timeouts: issue

The system auto-fixes predictable cases (dependency, lint) and files issues for the rest (tests, build config, timeouts).

How I Validate Locally (Fast Feedback Loop)

The fastest way to iterate is to run everything locally and forward GitHub webhooks into your laptop.

Backend:

cd /mnt/d/repos/pipelinehealer/backend
source .venv/bin/activate
python3 scripts/aoai_smoke.py
uvicorn src.main:app --reload --port 8000

Webhook forwarding (recommended for local dev):

bunx smee-client --url https://smee.io/<your-channel> --target http://127.0.0.1:8000/webhook/github

Trigger failures in the demo repo:

gh workflow run CI -R Canepro/pipelinehealer-demo -f failure_type=build_config

One concrete milestone from this loop: a failed run produced a real remediation issue in the demo repo (no manual triage).

What’s Next

Now that the pipeline works end-to-end locally, the next two phases are about demo safety:

• Security: protect /api/* (recommended: X-API-Key), and make webhook verification policy explicit.
• Reliability: add retries/backoff for GitHub API calls, and add timeouts per agent step.

Then comes the real deployment work: aligning the infrastructure and deploy path so the actual backend/frontend images can run in Azure reliably.

Checklist

When a system opens PRs on your behalf:

• Split responsibilities so each agent is narrow and testable.
• Use typed contracts; do not pass free-form text as your "API".
• Default to safe actions when uncertain.
• Add confidence gating and deterministic fallbacks.
• Make local E2E cheap (webhook forwarding + a demo repo).

Post 2 of 6 in the AI Dev Days Hackathon series.