kubernetes-operations

// Kubernetes cluster operations on minikube including observability (Grafana, Prometheus, Alertmanager, Loki, Tempo), debugging (kubectl debug, ephemeral containers), and cluster management (ArgoCD). Use when working with cluster/manifests/, Kubernetes workloads, pods, deployments, operators, controll

$ git log --oneline --stat

stars:2

forks:0

updated:March 2, 2026

SKILL.mdreadonly

SKILL.md Frontmatter

namekubernetes-operations

descriptionKubernetes cluster operations on minikube including observability (Grafana, Prometheus, Alertmanager, Loki, Tempo), debugging (kubectl debug, ephemeral containers), and cluster management (ArgoCD). Use when working with cluster/manifests/, Kubernetes workloads, pods, deployments, operators, controllers, or cluster components.

contextfork

keywordskubernetes, k8s, minikube, grafana, prometheus, loki, tempo, argocd, pod, クラスタ, 監視, kubectl, deployment

Access Grafana at https://grafana.minikube.127.0.0.1.nip.io
Disable ArgoCD selfHeal before manual changes, re-enable after

Debugging

Investigating	Tool	Entry Point
Traces, metrics, logs, profiles	Grafana	`https://grafana.minikube.127.0.0.1.nip.io`
Pod startup failures, events	kubectl	`kubectl get events -n <namespace>`
In-container process/network	kubectl debug	Ephemeral container

Observability Investigation (via Grafana)

Use Grafana for all observability signals. Do NOT query backends (Tempo, Mimir, Loki) directly.

Get query parameters - Check cluster/manifests/<app>/ for namespace, labels, OTEL_SERVICE_NAME (see Queries for parameter locations)
Open Grafana - Access https://grafana.minikube.127.0.0.1.nip.io in browser
Select datasource and query - Use appropriate signal:

Signal	Backend	Datasource	Query Language	Use Case
Traces	Tempo	Tempo	TraceQL	Request flow, latency, access destinations
Metrics	Mimir (Prometheus)	Mimir	PromQL	Resource usage, HTTP rates, alerting
Logs	Loki	Loki	LogQL	Error investigation, audit
Profiles	Pyroscope	Pyroscope	Flamegraph UI	CPU/memory hotspots
Probes	Blackbox Exporter	Mimir	PromQL	Endpoint reachability

Symptom	Signal	Query Approach
Errors in logs	Loki → Tempo	Extract traceid from logs, trace in Tempo
Latency/5xx	Tempo	Search traces with `status = error`
Unknown outbound dependencies	Tempo	Search traces, inspect spans for outbound calls
Resource saturation	Mimir	Query CPU/memory metrics
High CPU/memory	Pyroscope	Check flamegraphs

Pod Direct Investigation (via kubectl)

Use kubectl when Grafana cannot answer the question (e.g., pod not starting, container-level inspection).

Symptom	Action
Pod not starting	`kubectl get events -n <namespace>`
CrashLoopBackOff	`kubectl logs <pod> -n <namespace> --previous`
Network connectivity	`kubectl debug` with ephemeral container
Process inspection	`kubectl debug` with ephemeral container

Ephemeral Container

kubectl debug <pod-name> -n <namespace> \
  --profile=restricted \
  --image=ghcr.io/hippocampus-dev/hippocampus/ephemeral-container:main \
  --target=<container-name> \
  -- <command>

Note: Do not use -it flag when executing commands. It causes output streaming issues.

Manual Changes

Required when directly modifying live cluster resources (e.g., kubectl apply, kubectl patch, kubectl delete) outside of the GitOps workflow.

Operation	Requires Manual Changes
`kubectl apply/patch/delete`	Yes
Debugging (read-only: logs, events, debug)	No
Grafana queries	No
Editing manifests in repo	No (ArgoCD syncs automatically)

ArgoCD selfHeal Control

ArgoCD reverts manual changes unless selfHeal is disabled first. Always re-enable after.

# Disable selfHeal
kubectl patch application <app-name> -n argocd --type=merge \
  -p '{"spec":{"syncPolicy":{"selfHeal":false}}}'

# Re-enable selfHeal (after work is complete)
kubectl patch application <app-name> -n argocd --type=merge \
  -p '{"spec":{"syncPolicy":{"selfHeal":true}}}'

External Service Access

Services exposed via Istio Gateway follow a tiered host naming convention:

Host	Authentication	Access
`{service}.minikube.127.0.0.1.nip.io`	None	Local development
`{service}.kaidotio.dev`	OAuth2 (ext-authz)	Browser access
`{service}-public.kaidotio.dev`	Custom (JWT, method restriction, etc.)	Programmatic access excluded from OAuth2

The -public variant is used when OAuth2 is technically impossible (e.g., W3C Reporting API browser-generated requests) or when the service implements its own AuthorizationPolicy (e.g., GitHub Actions OIDC token validation).

To access a service's data locally, use https://{service}.minikube.127.0.0.1.nip.io.

Observability Stack Manifests

Component	Path
Grafana	`cluster/manifests/grafana/`
Tempo	`cluster/manifests/tempo/`
Mimir	`cluster/manifests/mimir/`
Loki	`cluster/manifests/loki/`
Pyroscope	`cluster/manifests/pyroscope/`
Prometheus	`cluster/manifests/prometheus/`
Fluentd	`cluster/manifests/fluentd/`
OpenTelemetry	`cluster/manifests/otel-agent/`, `cluster/manifests/otel-collector/`

Reference

If writing observability queries: See Queries