Deploying a Containerized Go App on Kubernetes

A Go binary is close to the ideal Kubernetes workload: one static file, fast startup, small memory footprint, no runtime to install. The manifests are correspondingly short.

What takes the time is the handful of fields that decide whether a deploy is invisible to users or drops a few hundred requests every time. Those are the parts this covers in detail.

Assumes an image built as in Docker containers for Go applications, and a cluster you can reach with kubectl. Written against Kubernetes 1.29 and Go 1.22.

The application needs two things first

Before any YAML, the binary has to cooperate with the platform in two ways.

A health endpoint that means something. Not return 200, a check that reflects whether this instance can serve:

func main() {
    mux := http.NewServeMux()

    // liveness: is the process itself wedged?
    mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })

    // readiness: can it serve traffic right now?
    mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
        ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
        defer cancel()
        if err := db.PingContext(ctx); err != nil {
            http.Error(w, "database unavailable", http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
    })

    srv := &http.Server{Addr: ":8080", Handler: mux}
    // ... see graceful shutdown below
}

The distinction is the single most consequential thing here. Liveness answers “should this container be killed and restarted”. Readiness answers “should traffic be sent to it”. Pointing a liveness probe at a database check means a database blip restarts every replica simultaneously: a self-inflicted outage, and a crash loop that keeps the service down after the database recovers.

Graceful shutdown. Kubernetes sends SIGTERM and waits:

    go func() {
        if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
            log.Fatalf("listen: %v", err)
        }
    }()

    stop := make(chan os.Signal, 1)
    signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
    <-stop

    log.Println("shutting down")
    ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
    defer cancel()
    if err := srv.Shutdown(ctx); err != nil {
        log.Printf("forced shutdown: %v", err)
    }

srv.Shutdown stops accepting new connections and lets in-flight requests finish. Without it every rollout kills requests mid-flight.

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes-api
  labels:
    app: notes-api
spec:
  replicas: 3
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: notes-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: notes-api
    spec:
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        fsGroup: 65532
      containers:
        - name: notes-api
          image: ghcr.io/example/notes-api:1.4.0
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: PORT
              value: "8080"
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: notes-secrets
                  key: db-password
          envFrom:
            - configMapRef:
                name: notes-config
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              memory: 128Mi
          startupProbe:
            httpGet: { path: /healthz, port: http }
            failureThreshold: 30
            periodSeconds: 1
          livenessProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet: { path: /readyz, port: http }
            periodSeconds: 5
            failureThreshold: 2
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

Several of those fields are load-bearing rather than decorative.

maxUnavailable: 0 means the rollout adds a new pod before removing an old one, so capacity never dips below replicas. The default is 25% unavailable, which on three replicas means serving with two during every deploy.

A pinned image tag, never :latest. With latest you cannot tell which version is running, kubectl rollout undo has nothing distinct to roll back to, and imagePullPolicy behaviour becomes surprising. Tag with the version or the commit SHA.

requests matter more than limits. Requests are what the scheduler uses to place the pod and what guarantees it capacity. A Go service that idles at 20m CPU and needs 200m under load should request something realistic: request too little and it is scheduled onto a full node and throttled.

No CPU limit here, deliberately. A CPU limit throttles via CFS quota, and a Go program with several goroutines can exhaust its quota slice and stall for the rest of the period even when the node is idle. Latency gets worse, not more predictable. Memory is limited, because memory is incompressible. Exceeding it should kill the pod rather than the node.

If you do set a CPU limit, set GOMAXPROCS to match it. Go reads the number of host cores, not the cgroup quota, so a container limited to 500m still starts 64 OS threads on a 64-core node — automaxprocs or an explicit env var fixes it.

startupProbe protects a slow start. Without it, a liveness probe with failureThreshold: 3 and periodSeconds: 10 kills anything taking over 30 seconds to boot, and then again, forever. The startup probe suspends liveness until the app is up. For a Go binary this is usually unnecessary; for one that warms a cache it is essential.

preStop: sleep 5 is not a hack. Pod termination and Service endpoint removal happen concurrently: the kubelet sends SIGTERM while kube-proxy is still updating iptables on every node. For a few hundred milliseconds to a few seconds, traffic still arrives at a pod that has begun shutting down. Sleeping before SIGTERM gives endpoint propagation time to finish, and it is the standard answer to “why do I see connection errors during a rollout when I have graceful shutdown”.

Note the ordering: preStop runs before SIGTERM, and terminationGracePeriodSeconds covers preStop plus shutdown together. So 5 + 25 must be under 30.

Service and Ingress

apiVersion: v1
kind: Service
metadata:
  name: notes-api
spec:
  type: ClusterIP
  selector:
    app: notes-api
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: notes-api
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: [api.example.com]
      secretName: notes-api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: notes-api
                port:
                  number: 80

targetPort: http by name rather than 8080 by number. Change the container port and the Service follows; a hardcoded number silently points at nothing, and the symptom is a Service with no working endpoints rather than an error.

ClusterIP plus an Ingress, not LoadBalancer per service: one load balancer for the cluster rather than one per service.

Configuration and secrets

apiVersion: v1
kind: ConfigMap
metadata:
  name: notes-config
data:
  LOG_LEVEL: info
  FEATURE_SEARCH: "true"
---
apiVersion: v1
kind: Secret
metadata:
  name: notes-secrets
type: Opaque
stringData:
  db-password: change-me

stringData rather than data so you write plaintext and Kubernetes does the base64. And be clear about what a Secret is: base64 is not encryption. Anyone with read access to Secrets in the namespace, or to etcd, can read it. Enable encryption at rest, restrict RBAC, and for anything serious use an external secret store.

A ConfigMap change does not restart pods. Environment variables are read once at container start, so a config update has no effect until a rollout:

$ kubectl rollout restart deployment/notes-api

Mounted-as-file ConfigMaps do update in place, after a delay, but only if your app re-reads the file.

Deploy and verify

$ kubectl apply -f k8s/
$ kubectl rollout status deployment/notes-api --timeout=120s
deployment "notes-api" successfully rolled out

$ kubectl get pods -l app=notes-api
NAME                         READY   STATUS    RESTARTS   AGE
notes-api-7d4b8c9f5-2xk4p    1/1     Running   0          40s
notes-api-7d4b8c9f5-8mn2q    1/1     Running   0          38s
notes-api-7d4b8c9f5-qw7rt    1/1     Running   0          35s

$ kubectl port-forward svc/notes-api 8080:80
$ curl -s localhost:8080/healthz

kubectl rollout status with a timeout is what belongs in CI. It exits non-zero if the rollout does not complete, which kubectl apply alone does not: apply succeeds the moment the API server accepts the object, whether or not a single pod ever becomes ready.

Where to look when something is wrong, in order:

$ kubectl describe pod <name>        # Events at the bottom: scheduling, pull, probe failures
$ kubectl logs <name> --previous     # the CRASHED container's logs, not the new one
$ kubectl get endpoints notes-api    # empty means no pod is Ready or the selector is wrong

--previous is the one people forget. After a CrashLoopBackOff the current container has barely started; the reason is in the previous one’s logs.

An empty endpoints list with running pods means either the readiness probe is failing or the Service selector does not match the pod labels. Those are the two causes, and describe distinguishes them.

Scaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: notes-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: notes-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

averageUtilization is a percentage of the request, not of the node or the limit, so 70% of a 50m request is 35m, which is a very low bar. Autoscaling only behaves sensibly when requests reflect real usage, which is another reason to measure rather than guess them.

An HPA needs metrics-server installed, and it will fight a hardcoded replicas in your Deployment. Remove that field once an HPA owns the count, or leave it and accept that each apply resets scale.

Add a PodDisruptionBudget so voluntary disruptions: a node drain, a cluster upgrade — cannot take everything at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: notes-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: notes-api

Frequently asked questions

Why do I see errors during a rollout even with graceful shutdown?

Endpoint removal and SIGTERM happen concurrently, so traffic can still arrive at a terminating pod. Add a preStop sleep of a few seconds to let endpoint propagation finish first.

Should liveness and readiness use the same endpoint?

No. Liveness restarts the container; readiness only removes it from the Service. A liveness probe that checks a database restarts every replica when the database blips.

Why is my pod in CrashLoopBackOff?

Read kubectl logs <pod> --previous, the current container has just started, and the reason is in the one that died.

Why does my Service have no endpoints?

Either no pod is passing its readiness probe, or the Service selector does not match the pod labels. kubectl describe tells you which.

Should I set a CPU limit?

Usually not for a Go service. CFS throttling can stall goroutines for the rest of a period even on an idle node. Always set a memory limit, since memory is incompressible.

Do I need to set GOMAXPROCS?

If you set a CPU limit, yes. Go reads host cores, not the cgroup quota, so a 500m-limited container otherwise starts threads for every core on the node. automaxprocs handles it automatically.

Why is my ConfigMap change not taking effect?

Environment variables are read once at container start. Run kubectl rollout restart, or mount the ConfigMap as a file and re-read it.

Are Kubernetes Secrets encrypted?

No. They are base64-encoded by default. Enable encryption at rest, restrict RBAC, and use an external secret store for anything sensitive.

Why did apply succeed but nothing works?

apply only confirms the API server accepted the object. Use kubectl rollout status --timeout=…, which fails when pods never become ready.

What does averageUtilization 70 actually mean?

70% of the CPU request, not of the node or the limit. Unrealistic requests make an HPA scale at the wrong time.

Where should I go next?

Docker containers for Go applications covers building the image this deploys, and the DevOps guides cover the rest of the pipeline.