Cloud
Central Observability Hub (Terraform-provisioned OKE + ArgoCD-managed LGTM)
Terraform-provisioned OKE hub running an ArgoCD-managed LGTM stack (Grafana/Loki/Tempo) with OTLP ingestion and durable storage backends.
Historical deployment. The hosted environment has been retired; source and recorded evidence remain available.
Kubernetes · OKE · Grafana · Prometheus · Loki · Tempo · NGINX Ingress · cert-manager · Terraform · ArgoCD
Overview
Centralized observability hub deployed on Oracle Kubernetes Engine (OKE) to aggregate metrics, logs, and traces from multiple environments into one place.
Deployment status
The hosted Grafana environment has been retired. Source and recorded dashboards remain as evidence of the build.
Stack
- Grafana (dashboards / exploration)
- Prometheus (metrics collection + alerting patterns)
- Loki (log aggregation)
- Tempo (distributed tracing)
- Alertmanager (alert routing)
- NGINX Ingress + cert-manager (TLS + ingress routing)
Security notes
- TLS terminated at ingress using cert-manager (Let's Encrypt).
- Ingestion endpoints protected with Basic Auth (no anonymous remote-write / push endpoints).
- Internal services remain ClusterIP unless explicitly exposed through authenticated ingress.
Storage
- Object storage for Loki/Tempo persistence (OCI Object Storage / S3-compatible backend)
SRE notes
- One hub aggregates telemetry from multiple clusters
- Metrics, logs, and traces correlate in one place
- Ingestion endpoints require auth; internal services stay ClusterIP unless exposed through ingress
Outcomes
Grafana/Loki/Tempo hub on OKE with authenticated ingestion, object storage, and Terraform + Argo CD deploy.
Challenges and solutions
Challenges
- Designing a single hub to aggregate telemetry from multiple clusters/environments safely
- Exposing ingestion endpoints publicly without allowing anonymous writes
- Balancing cost constraints (free tier budget) with operational reliability
- Choosing storage patterns that scale beyond local PVCs for logs and traces
- Operating the stack with reproducible change control (Terraform + ArgoCD) rather than manual drift
Solutions
- Deployed Grafana/Prometheus/Loki/Tempo on OKE with NGINX Ingress and cert-manager TLS
- Implemented Basic Auth for ingestion endpoints to prevent unauthenticated writes
- Kept internal components as ClusterIP and only exposed required ingress routes
- Backed Loki/Tempo with object storage for durable, scalable persistence
- Used Terraform for OCI/OKE lifecycle and ArgoCD for continuous reconciliation of the observability stack
Technology stack
Kubernetes
OKE · NGINX Ingress · cert-manager · Helm
Observability
Grafana · Prometheus · Loki · Tempo · Alertmanager
Security
TLS · Basic Auth · ClusterIP services
Storage
OCI Object Storage (S3-compatible)