Skip to content
Vincent Mogah
Email Vincent
Back to writing

Deploying PipelineHealer to Azure: What Went Wrong (and Right)

Local was easy. Azure was where the real systems work started: ACR auth, Container Apps revisions, webhook signatures, and production-safe defaults.

hackathon · azure · container-apps · ci-cd · devops · github-actions

Local dev hid gaps that showed up immediately in Azure:

  • identity and registry permissions
  • runtime config drift
  • webhook authenticity
  • frontend-to-backend proxy assumptions
  • scale-to-zero and revision health on Container Apps

Below: what failed and what fixed it.

Target Architecture

I kept the deployment intentionally simple:

  • Azure Container Apps for backend + frontend
  • Azure Container Registry (ACR) for images
  • Azure OpenAI for model inference
  • Azure Cosmos DB for activity storage
  • Key Vault + App Insights for secrets and telemetry

For this demo stage, I used a PAT in backend runtime for GitHub API calls and enforced API/webhook controls in production mode.

Failure 1: Provisioning Blocked by Key Vault Name Rules

Initial azd provision failed on Key Vault naming validation. The generated name exceeded constraints.

Fix:

  • Updated Bicep naming strategy to stay within Key Vault limits (3-24 chars, alphanumeric/hyphen rules).
  • Re-ran provisioning with shorter deterministic names.

Naming limits block deploys; treat them as hard constraints.

Failure 2: Backend Container App Revision Timed Out

azd provision kept failing with:

  • ContainerAppOperationError
  • Operation expired

Root cause was image pull permissions from ACR.

Fix:

  1. Added a user-assigned managed identity for ACR pull.
  2. Granted AcrPull role assignment on ACR scope.
  3. Updated Container App registry config to use that identity.
  4. Re-provisioned.

After this, backend and frontend container apps provisioned cleanly.

Operation expired often means ACR pull auth is wrong, not just a timeout.

Failure 3: Frontend Looked Up But API Proxy Broke

Frontend initially loaded inconsistently and /api/* behavior was unstable in Azure.

Root causes:

  • Nginx proxy assumed static backend host.
  • Host/SNI forwarding was incomplete.
  • Backend API auth needed X-API-Key, but frontend proxy did not inject it.

Fix:

  • Switched frontend container to runtime-templated Nginx config.
  • Added entrypoint.sh to render backend upstream dynamically.
  • Forwarded Host correctly and enabled SNI for TLS upstream.
  • Injected X-API-Key from frontend runtime env for /api/*.

Render Nginx config at container start; do not bake backend host assumptions into the image.

Failure 4: Webhooks Delivered But No Activities

I had valid CI failures in GitHub, but dashboard activity list was empty.

Root causes:

  • Multiple active hooks (legacy smee.io + Azure hook) caused confusion.
  • Azure hook secret mismatch produced delivery 401 responses.

Fix:

  • Disabled stale smee hook for Azure-mode testing.
  • Kept one active workflow_run hook pointing directly to Azure backend.
  • Re-synced GITHUB_WEBHOOK_SECRET between GitHub hook and backend runtime.
  • Verified delivery ping/status 200.

One environment, one active hook. Extra hooks create ambiguous signal flow.

Failure 5: Dashboard Stats Endpoints Failed in Cloud

After deployment, dashboard charts showed loading/errors while raw activities existed.

Root cause:

  • Async Cosmos path hit an unsupported query kwarg (enable_cross_partition_query) in this runtime path.

Fix:

  • Hardened backend aggregation logic for stats/failure-breakdown.
  • Removed unsupported async query option in affected paths.
  • Verified /api/stats and /api/failure-breakdown from frontend route and direct API.

Local storage mode passing does not guarantee the production Cosmos query path.

Failure 6: PR Creation 422 on Repeated Demo Runs

Dependency/lint remediation sometimes failed with:

  • 422 Unprocessable Entity on POST /repos/.../git/refs

Root cause:

  • Target remediation branch names already existed from prior runs (fix/update-left-pad, fix/lint-eslint-config).

Fix:

  • Merge/close old fix PRs before rerunning same scenario.
  • Treat as expected duplicate-run behavior in demo reset flow.

Branch naming needs an idempotency plan for reruns.

Post-Publish Updates (Feb 12, evening)

After publishing this post, I shipped a few meaningful reliability upgrades worth capturing here.

1) One-command operator workflow

I added a single entrypoint script for daily operations:

bash scripts/ph.sh help

This now wraps deploy/demo/scale/status actions without bouncing between multiple runbooks:

  • deploy (full rebuild + push + container app update + verification)
  • deploy:env (runtime env updates only)
  • demo:e2e, demo:reset
  • warm, lowcost, status
  • settings:check

2) Admin settings flow and auth validation

The new admin settings flow is now practical in deployed environments:

  • Settings endpoint and page are protected by admin auth.
  • Deploy verification now checks both API and admin headers.
  • Runtime values can be confirmed quickly with:
bash scripts/ph.sh settings:check

3) WSL/Podman deploy script hardening

I fixed two real operator pain points:

  • Guarded against accidentally sourcing deploy scripts (which could terminate shell sessions).
  • Normalized compose env-file paths for Podman-on-WSL external compose providers.

This removed a class of "script ran locally but failed in shell/runtime boundary" issues.

4) Azure OpenAI compatibility correction

A subset of runs failed diagnosis with 400 API version not supported.

Fix:

  • Aligned runtime to a supported API version for the active endpoint.
  • Re-deployed env config and re-verified from live settings.

This restored stable diagnosis behavior for non-deterministic paths.

What Worked Well

  • Once identity + proxy + webhook controls were correct, the system stabilized quickly.
  • Production security defaults held:
    • API key required for /api/*
    • webhook signature verification enabled
  • Azure OpenAI deployment switch to gpt-5-mini was straightforward after config alignment.

Operational Pattern I Will Keep

I now use a warm/low-cost toggle for demo reliability:

  • warm mode: min-replicas=1 on backend/frontend during demos
  • low-cost mode: min-replicas=0 after demos

This keeps spend low while avoiding cold-start surprises during recordings.

Takeaway

Deploy early if the product runs on cloud infra. Identity, webhook verification, runtime config injection, and idempotent branch naming are all product behavior, not side quests.

Local proved the idea. Azure proved the wiring.

Post 3 of 6 in the AI Dev Days Hackathon series.