Skip to content
Vincent Mogah
Email Vincent
Back to writing

Moving Rocket.Chat to a Personal AKS Cluster Without Breaking the GitOps Loop

Rocket.Chat migration to a new AKS cluster with Argo CD on OKE, Terraform remote state, Jenkins validation, and DNS cutover last.

kubernetes · gitops · argocd · terraform · aks · jenkins · migration

This migration moved Rocket.Chat from an older AKS cluster to a new personal one while keeping Argo CD and Jenkins on OKE, preserving the database, and cutting DNS only after the new stack was healthy.

Related project pages: Rocket.Chat on Kubernetes, GitOps migration, hybrid control plane.

The Intended End State

The shape I wanted was simple:

  • OKE stays the control plane
    • ArgoCD stays there
    • Jenkins controller stays there
    • multi-cluster orchestration stays centralized
  • AKS becomes the runtime
    • Rocket.Chat runs there
    • MongoDB runs there
    • Traefik, External Secrets, and the shared ops stack run there
    • the static Jenkins validation agent also runs there
  • Terraform state moves to Azure Storage, separate from the AKS workload resource group

Control plane stays stable. Workload cluster stays replaceable. Terraform state does not live next to the cluster it manages.

What Had to Be True Before Cutover

There were four non-negotiable pieces.

1. Terraform state had to stop being local

Before the migration could touch identities, Key Vault, schedules, or cluster configuration, the main stack had to be on a real remote backend:

  • resource group: rg-canepro-tfstate
  • storage account: caneprotfgmhl5a
  • container: tfstate
  • key: aks.terraform.tfstate

This was the first move because every other step depended on it being safe to collaborate on state and impossible to trample it accidentally.

2. ArgoCD had to target the new AKS cluster explicitly

The first operator trap was familiar:

  • stale DNS
  • a reused kube context name
  • two AKS clusters with similar enough identities to be confusing under pressure

The fix was not clever. I stopped treating kube context names as identity and used API server URLs as identity instead. The new AKS cluster was registered separately in ArgoCD, application destinations were updated deliberately, and the old cluster stayed intact until the new one was verified.

That preserved rollback options instead of gambling them away early.

3. The database had to move after the platform, not during it

The safe sequence was:

  1. get the new AKS platform healthy
  2. get operator-managed MongoDB running
  3. restore the rescued Rocket.Chat data
  4. restart Rocket.Chat against the restored database
  5. validate real application behavior
  6. only then move traffic

That discipline mattered. Mixing cluster migration, data restore, and application version change in the same step is how “simple” migrations become postmortems.

4. Jenkins had to be treated as part of the migration

I kept the controller on OKE, but the static validation agent moved to AKS. That meant the new cluster was not considered “done” until Jenkins PR validation was running from it.

That turned out to be the right call, because CI immediately started exposing real infra defects:

  • missing Azure RBAC for the workload identity
  • stale backend handling in the Jenkinsfile
  • Terraform budget validation edge cases
  • schedule reconciliation bugs that had been hiding in plain sight

CI was not an extra. It was part of proving the new runtime was real.

The Most Useful Failure: GitOps Exposed Drift

The rebuild surfaced drift that GitOps is meant to catch: resources that existed in the live cluster but not in Git.

MongoDB was the clearest case. The operator deployed, but Rocket.Chat failed because MongoDBCommunity was missing from the active Git-managed app path. The app tried to connect to a service that did not exist yet.

Repair: add the MongoDB resource to Git, fix namespace ordering, remove a dead cert-manager blocker from the ops app, re-sync.

The Data Restore Was Boring, Which Is the Goal

After the platform shape was right, the database move was intentionally boring.

MongoDB came up on the new cluster, the rescued dump restored cleanly, and Rocket.Chat recovered on restart. I verified that the data was actually present before moving on:

  • users
  • rooms
  • messages

If a stateful migration step is boring, that is usually success.

The Subtle Bug Was the Automation Schedule

The most interesting Terraform bug was not about compute, networking, or identity. It was about time.

The intended AKS schedule was:

  • start: 13:30 Europe/London
  • stop: 16:15 Europe/London
  • weekdays only

But Azure was still running the old window. The root cause was a classic “stability hack becomes a correctness bug” problem:

  • start_time had originally been seeded from timestamp()
  • Terraform was told to ignore changes to that field
  • that prevented daily plan churn
  • it also prevented legitimate time updates from ever reconciling

The fix was to replace the unstable seed with a stable time_static anchor and let real schedule changes flow through normally.

Suppressing drift in Terraform is not the same as modeling schedule intent correctly.

DNS Was the Final Step

Only after the following were true did I move traffic:

  • ArgoCD healthy on the new cluster
  • Rocket.Chat healthy on the new cluster
  • Mongo restored
  • Jenkins validation running from AKS
  • Terraform state and automation schedules reconciled

Then the DNS change was simple:

  • point k8.canepro.me at the new Traefik public IP
  • confirm public resolution
  • confirm HTTP 200
  • retire the old AKS registration afterward

DNS is the last step, not the plan.

What stuck

  1. Rebuilds expose drift. If Git cannot recreate the cluster, that is useful information.
  2. Cluster identity is the API server URL, not the kube context name.
  3. Run Jenkins validation from the new runtime before calling the migration done.
  4. Keep control plane, workload cluster, and Terraform state separate for safer rollback.
  5. Standing the environment up again is the fastest audit of declarative truth.

Final state

  • Rocket.Chat live on the new AKS cluster
  • OKE still runs Argo CD and Jenkins
  • Jenkins PR validation runs from the AKS static agent
  • Terraform uses the Azure remote backend
  • Weekday auto-start/stop matches the intended schedule
  • Stale migration dumps and cluster-specific leftovers removed

The cutover worked because Git, Terraform, Argo CD, Jenkins, and DNS moved in the same direction.