Homelab / Prometheus, Loki, and Grafana outside the cluster they watch.
Observability
The metrics and logs stack runs on its own guest and its interface is not served through the cluster: a dashboard that depends on the cluster is unavailable exactly when it is needed. Memory is budgeted rather than overcommitted; CPU is overcommitted on purpose, because intermittent workloads tolerate that and memory does not.
Prometheus collects metrics by scraping and Loki concentrates the logs.
Grafana queries both from its own guest, outside the cluster it watches.
Its interface resolves through the tunnel and does not cross the cluster Gateway.
How I built it
A dashboard served through the cluster stops being available exactly when it has to be read, so the observability stack does not share fate with the workloads it measures. Node memory is budgeted rather than overcommitted; CPU is overcommitted, because an intermittent workload tolerates that and memory does not.