PipelineHealer started as my entry for the AI Dev Days Hackathon. The most useful decision was not the model or cloud service. It was keeping the model out of the first layer.
That matters once the product touches real repositories.
PipelineHealer ingests failed CI runs, normalizes the evidence, diagnoses likely root causes, and takes a policy-safe remediation path. In practice, that means deterministic cases can open or reuse fix PRs, while ambiguous or risky cases fall back to structured issues with explicit reasoning. That boundary ended up mattering more than any prompt tweak.
This post is the durable version of a lesson I recently published as an X Article. It is grounded in the full repo history, not just the earlier hackathon posts. The original submission baseline was v0.3.0 on March 2, 2026. The current public baseline is v0.7.2 from March 13, 2026. The product kept evolving after the submission cut, but this design rule held up the entire time.
If you want the earlier project context, start with Post 1, the multi-agent architecture post, the Azure deployment lessons, the log parsing write-up, the webhook-to-PR flow, and the hackathon retrospective.
For the public project itself, see the PipelineHealer repository and the architecture diagram.
The Obvious Hackathon Architecture Was the Wrong One
The natural instinct in an AI hackathon is to make the model the center of everything.
That was my first instinct too. The early versions looked like this:
- collect a failed CI log
- send a large chunk of it to the model
- ask for a diagnosis
- ask for a fix
It produced interesting output. It did not produce a foundation I would trust.
The problem was not that the model was useless. The problem was that a model-first flow was weak in exactly the places a remediation system cannot be weak:
- repeatability
- testability
- safety boundaries
- operator trust
If a tool is going to open PRs or suggest changes against a real repository, "it sounded plausible in the demo" is not a good enough standard.
What Changed When I Flipped the Design
PipelineHealer got better when I changed the order:
- deterministic pattern matching first
- structured model fallback only when ambiguity remained
That decision changed the whole product.
Common cases such as dependency failures, lint and formatting errors, and some build configuration issues no longer needed a model call. They could be classified from the evidence and routed through explicit logic.
The model still mattered. It just mattered in the right place.
That gave me several things the model-first version could not provide reliably:
- faster handling for common failures
- unit-testable pattern logic
- clearer action boundaries
- lower risk when the system was allowed to create artifacts
- better operator explanations when the system did escalate to AI help
This was not "less AI." It was a better contract for where AI added value.
Why This Was the Safety Decision, Not Just the Architecture Decision
Pattern-first handling changed the trust model as much as the technical design.
With deterministic-first classification, I could keep the default posture conservative:
- low-confidence outcomes create issues instead of PRs
- review-first paths stay explicit
- repository allowlists gate real-repo rollout
- remediation is tied back to concrete failure evidence
That meant the system could still be ambitious without behaving recklessly.
During the hackathon, I never had a moment where PipelineHealer did something surprising to a repository. That was not luck. It came from deciding early that deterministic evidence should be the first layer and AI should be a bounded capability inside the control path, not the whole control path.
The Real Product Work Was Around the Agent Logic
By the end of the hackathon, PipelineHealer was a deployed Azure system with GitHub webhook intake, diagnosis, fix PR or issue generation, a dashboard, and a one-command operator CLI.
The important thing I learned is that the model logic was only part of the product.
Some of the hardest work was elsewhere:
- deploying to Azure Container Apps early enough to find identity and proxy failures
- handling late-arriving external diagnostics and adding async backfill
- sanitizing third-party AI output before showing it in the UI
- giving operators one consistent entrypoint through
bash scripts/ph.sh - making the runtime posture and governance surfaces honest instead of aspirational
That is why I do not think of PipelineHealer as "a prompt wrapped in a dashboard." The surrounding engineering made the model behavior usable.
The Submission Baseline Was Real, but the Product Did Not Stop There
The repository history matters here because it proves this was not just a submission-week story.
The hackathon submission baseline was v0.3.0, which shipped the Activity Detail Copy Context handoff flow and a visible disabled Assign to Agent affordance for discoverability. After that baseline, the project kept moving:
v0.3.2added Jenkins bridge support, Assign-to-Agent activation, and storage posture hardeningv0.4.0reframed PipelineHealer as an OSS-first remediation control plane rather than a hackathon-only CI demov0.5.0added outbound integration and notification routingv0.6.0added typed diagnosis and remediation hardening, plus operator trust workflowsv0.7.0andv0.7.2tightened runtime configuration, release discipline, and Jenkins-side diagnosis reliability
That release trail matters because the original pattern-first decision survived all of it.
It held up when the product became more operational, more governed, and more honest about its runtime boundaries.
Why This Is Also Shaping SignalForge
This lesson is not staying inside PipelineHealer.
SignalForge is the diagnostics sibling to PipelineHealer. Current scope:
- ingest infra artifacts
- produce structured findings
- one optional LLM pass for explanation and ranking
- no remediation yet
PipelineHealer handles write paths. SignalForge stays read-only until the safety model is clear. Same lesson, narrower boundary.
What I Would Reuse
Same rule for the next ops tool with AI in the loop:
Pattern matching first. LLM only when the failure does not match a known class.
That keeps the system easier to test, explain, and roll back.
Reusable Takeaway Checklist
- Define what the model cannot do before wiring what it can.
- Extract structured evidence before free-form diagnosis.
- Treat low-confidence outcomes as review tasks, not auto-fix wins.
- Build the CLI/runbook path early; it surfaces gaps faster than UI polish.
- Deploy to the real environment early enough to catch identity and integration failures.
- Keep read-only diagnostics separate from write-capable remediation.
That rule held from hackathon submission through v0.7.2.
Post 7 of 7 in the AI Dev Days Hackathon series.