Deploying a Go App with Redis on Kubernetes

Deploying a stateless Go service is covered in Deploying a containerized Go app on Kubernetes. This is about what changes when it depends on Redis, which is where the interesting failures are, because two of them look like Kubernetes bugs and are configuration.

Written against Kubernetes 1.29, Go 1.22 and Redis 7.2.

First: is this Redis a cache or a database?

The answer decides everything else, and it is worth answering explicitly rather than by default.

A cache. Losing it costs latency, not data. Deploy it as a Deployment with no volume, cap its memory, and let it evict. Your application must work with every key missing.

A datastore. Losing it loses data: session state you cannot rebuild, a queue, a counter that matters. That needs a StatefulSet, a persistent volume, and a considered answer about replication.

Most Redis deployments are the first while being configured like neither: a Deployment with a persistent volume attached, which is the specific combination that breaks.

The trap: a Deployment with a ReadWriteOnce volume

# this deadlocks on every rollout
apiVersion: apps/v1
kind: Deployment
spec:
  strategy:
    type: RollingUpdate         # the default
  template:
    spec:
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: redis-data    # ReadWriteOnce

A RollingUpdate starts the new pod before terminating the old one. A ReadWriteOnce volume can only be mounted by one node at a time, so the new pod sits in ContainerCreating waiting for a volume the old pod still holds, and the old pod is not terminated until the new one is ready.

Nothing errors. The rollout hangs, kubectl describe pod shows a FailedAttachVolume or Multi-Attach error event, and it stays that way until you intervene.

Two correct configurations:

# a datastore: StatefulSet, which replaces pods one at a time
apiVersion: apps/v1
kind: StatefulSet

# or, if it must be a Deployment with a volume: Recreate
spec:
  strategy:
    type: Recreate      # terminate the old pod first, accept the downtime

Recreate means a gap with no Redis during every deploy, which is acceptable for a cache and not for a datastore. That is the trade the strategy field is making explicit.

Redis as a cache

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-config
data:
  redis.conf: |
    maxmemory 200mb
    maxmemory-policy allkeys-lru
    appendonly no
    save ""
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels: { app: redis }
  template:
    metadata:
      labels: { app: redis }
    spec:
      containers:
        - name: redis
          image: redis:7.2-alpine
          args: ["redis-server", "/etc/redis/redis.conf", "--requirepass", "$(REDIS_PASSWORD)"]
          ports:
            - name: redis
              containerPort: 6379
          env:
            - name: REDIS_PASSWORD
              valueFrom:
                secretKeyRef: { name: redis-secret, key: password }
          volumeMounts:
            - name: config
              mountPath: /etc/redis
          resources:
            requests: { cpu: 50m, memory: 256Mi }
            limits:   { memory: 300Mi }
          livenessProbe:
            exec:
              command: ["redis-cli", "ping"]
            initialDelaySeconds: 10
          readinessProbe:
            exec:
              command: ["sh", "-c", "redis-cli -a \"$REDIS_PASSWORD\" ping"]
            periodSeconds: 5
      volumes:
        - name: config
          configMap: { name: redis-config }
---
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  clusterIP: None            # headless: DNS resolves straight to the pod
  selector: { app: redis }
  ports:
    - port: 6379
      targetPort: redis

Three numbers in there are related and easy to get wrong.

maxmemory must sit below the container’s memory limit. Redis counts its dataset, not its total RSS: fragmentation, client buffers and copy-on-write during a background save all live outside that figure. maxmemory 200mb under a 300Mi limit leaves headroom. Set them equal and the kernel OOM-kills the container before Redis ever evicts a key, which presents as Redis “randomly restarting”.

maxmemory-policy allkeys-lru is what makes it a cache. The default is noeviction, which means Redis starts returning OOM command not allowed on writes once full rather than making room. For a cache that is exactly wrong.

appendonly no and save "" disable both persistence mechanisms. A cache that forks to write an RDB snapshot doubles its memory during the fork, which is a good way to be OOM-killed for no benefit.

The readiness probe authenticates. With requirepass set, an unauthenticated redis-cli ping returns NOAUTH, so the probe fails and the pod never becomes ready, a self-inflicted outage that looks like Redis not starting.

Redis as a datastore

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: redis          # must match the headless Service
  replicas: 1
  selector:
    matchLabels: { app: redis }
  template:
    metadata:
      labels: { app: redis }
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: redis
          image: redis:7.2-alpine
          args: ["redis-server", "/etc/redis/redis.conf"]
          ports:
            - name: redis
              containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
            - name: config
              mountPath: /etc/redis
          resources:
            requests: { cpu: 100m, memory: 1Gi }
            limits:   { memory: 1500Mi }
      volumes:
        - name: config
          configMap: { name: redis-config }
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 8Gi

volumeClaimTemplates gives each replica its own PVC, named data-redis-0. That is the difference from a Deployment: identity and storage are stable across restarts, and the replacement pod gets the same volume.

For persistence, the config changes:

appendonly yes
appendfsync everysec
maxmemory-policy noeviction

noeviction here, deliberately, a datastore must not silently discard data to make room. Running out of memory should be an error you notice, not data that quietly disappears.

terminationGracePeriodSeconds: 60 gives Redis time to flush its append-only file on shutdown.

Deleting a StatefulSet does not delete its PVCs. That is a safety feature and a surprise: recreate the StatefulSet and it adopts the existing volumes, including data you thought was gone.

One thing worth stating plainly: a single Redis pod is a single point of failure, whatever the workload type. Node failure means an outage until it reschedules, plus however long the AOF takes to load. If that is unacceptable, you need Sentinel, Redis Cluster, or a managed service: not a bigger replicas number, which for a StatefulSet gives you several independent Redis instances rather than one replicated one.

The Go side

package cache

import (
    "context"
    "errors"
    "time"

    "github.com/redis/go-redis/v9"
)

type Cache struct {
    client *redis.Client
}

func New(addr, password string) *Cache {
    return &Cache{client: redis.NewClient(&redis.Options{
        Addr:            addr,
        Password:        password,
        DB:              0,
        PoolSize:        10,
        MinIdleConns:    2,
        DialTimeout:     2 * time.Second,
        ReadTimeout:     500 * time.Millisecond,
        WriteTimeout:    500 * time.Millisecond,
        MaxRetries:      2,
        MinRetryBackoff: 20 * time.Millisecond,
    })}
}

// Get returns the value, or ok=false for both a miss and an outage.
// The caller cannot tell them apart, and must not need to.
func (c *Cache) Get(ctx context.Context, key string) (string, bool) {
    v, err := c.client.Get(ctx, key).Result()
    if err == nil {
        return v, true
    }
    if !errors.Is(err, redis.Nil) {
        // a real failure, not a miss
        metrics.CacheErrors.Inc()
    }
    return "", false
}

func (c *Cache) Set(ctx context.Context, key, value string, ttl time.Duration) {
    if err := c.client.Set(ctx, key, value, ttl).Err(); err != nil {
        metrics.CacheErrors.Inc()      // log and continue; never propagate
    }
}

Three deliberate choices.

Timeouts in the hundreds of milliseconds. Redis operations are sub-millisecond in the normal case, so a 500 ms read timeout is generous. Without one, a hung Redis holds every request that touches the cache, the cache becomes the thing that takes your service down, which is the opposite of its purpose.

redis.Nil is a miss, not an error. Treating it as a failure inflates your error metrics and triggers alerts on normal operation. Everything else genuinely is a failure.

A cache failure never propagates. Get returns ok=false for both a miss and an outage; Set swallows its error. The caller falls through to the database either way. That is the whole reason to treat this as a cache in the first place, and it has to be true in the code, not just in the intent.

Which leads to the probe:

// readiness must NOT include Redis for a cache-only dependency
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
    if err := db.PingContext(r.Context()); err != nil {   // the database is required
        http.Error(w, "database unavailable", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)                          // Redis is not
})

Putting Redis in the readiness probe means a Redis restart removes every application pod from the Service simultaneously: a full outage caused by a component you deliberately made optional. Report its health separately, on a metric or a /debug endpoint, and alert on that.

Wiring it up

env:
  - name: REDIS_ADDR
    value: redis:6379
  - name: REDIS_PASSWORD
    valueFrom:
      secretKeyRef: { name: redis-secret, key: password }

redis:6379, the Service name, resolved by cluster DNS. The full form is redis.default.svc.cluster.local; the short name works within the namespace.

No init container waiting for Redis. The client retries with backoff, and the application is designed to work without the cache, so blocking startup on it would reintroduce the coupling you just removed. An init container waiting for the database is defensible; for a cache it is not.

Restrict who can reach it:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: redis-allow-api
spec:
  podSelector:
    matchLabels: { app: redis }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: notes-api }
      ports:
        - port: 6379

Without a policy, every pod in the cluster can reach Redis on 6379. requirepass is one layer; a NetworkPolicy is the one that stops a compromised pod elsewhere from even connecting. Note that it requires a CNI that implements policies: on one that does not, this object is accepted and enforces nothing, which is worth verifying rather than assuming.

Checking it

$ kubectl get pods -l app=redis
NAME                     READY   STATUS    RESTARTS   AGE
redis-0                  1/1     Running   0          2m

$ kubectl exec -it redis-0 -- sh -c 'redis-cli -a "$REDIS_PASSWORD" info memory | head -5'
used_memory_human:1.05M
maxmemory_human:200.00M
maxmemory_policy:allkeys-lru

$ kubectl exec -it deploy/notes-api -- wget -qO- localhost:8080/readyz

maxmemory_policy in that output is the check worth doing after any config change, a ConfigMap edit does not restart the pod, so the running Redis may still be using the old policy. kubectl rollout restart is what applies it.

To confirm the cache is optional rather than merely described as optional, delete it and watch:

$ kubectl delete pod -l app=redis
$ kubectl logs -l app=notes-api --tail=20

Requests should keep succeeding, more slowly. If they fail, the coupling is real and the design is not.

Frequently asked questions

Why is my rollout stuck in ContainerCreating?

A Deployment with a ReadWriteOnce volume and a rolling update: the new pod waits for a volume the old pod still holds. Use a StatefulSet, or strategy: Recreate.

StatefulSet or Deployment for Redis?

Deployment for a pure cache with no volume. StatefulSet when data must survive a restart. It gives stable identity and a per-replica PVC.

Why does Redis keep getting OOM-killed?

maxmemory is at or above the container limit. Redis counts its dataset, not fragmentation, client buffers or fork overhead. Leave headroom.

Why is Redis refusing writes with OOM command not allowed?

maxmemory-policy is noeviction, the default. For a cache use allkeys-lru; for a datastore keep noeviction and treat it as an alert.

Why is my Redis pod never becoming ready?

The readiness probe is not authenticating. With requirepass, an unauthenticated redis-cli ping returns NOAUTH and the probe fails.

Should Redis be in my app’s readiness probe?

Not for a cache. A Redis restart would remove every application pod from the Service at once. Only required dependencies belong in readiness.

Do I need an init container to wait for Redis?

No. The client retries, and a cache-optional application should start without it. Waiting reintroduces the coupling.

Does deleting a StatefulSet delete its data?

No. PVCs created from volumeClaimTemplates survive, and a recreated StatefulSet adopts them.

How does my app find Redis?

By Service name, redis:6379, through cluster DNS. A headless Service resolves straight to the pod IP.

Is one Redis pod enough?

It is a single point of failure regardless of workload type. Increasing replicas on a StatefulSet gives independent instances, not replication, use Sentinel, Cluster, or a managed service.

Where should I go next?

Deploying a containerized Go app on Kubernetes covers the application manifests this builds on, and Docker containers for Go covers the image.