Skip to content

Receipts

Anyone can write “optimised Kubernetes” on a CV.

So here are the actual lines I changed, from the actual files. Client names, account IDs and certificate ARNs are cut out and each file is trimmed to the part that matters. Including the line I wrote that did nothing at all.

6 receipts · +22 -5 · 1 line I am not proud of

01 · Consumer travel platform · EKS · 6 Redis pods

Redis was told it could have more memory than it asked for

Found by lining up every memory setting in the cluster against the container it runs in.

production-manifests.yaml+2 -2
  env:
removed:     REDIS_ARGS: "--appendonly yes --maxmemory 3gb --maxmemory-policy allkeys-lru"
added:     REDIS_ARGS: "--appendonly yes --maxmemory 1536mb --maxmemory-policy allkeys-lru"
  resources:
    requests:
      memory: 2Gi
    limits:
removed:       memory: 4Gi
added:       memory: 2Gi

The scheduler packs nodes by requests, not limits. Each Redis pod asked for 2 GB and was then allowed to fill 3 GB of cache, plus whatever the AOF rewrite fork copies on top. Six of those on a busy node, and the memory everybody was counting on is not there. The kernel picks a victim and it is usually the biggest one.

Now maxmemory sits 512 MB under the limit, and request equals limit, so what the scheduler adds up is what actually gets used.

proof$ kubectl top pods -l app=redisevery pod under 2Gi, with room for the fork

02 · Same cluster · Elasticsearch · 3 pods

A 4 GB Java heap inside a 3 GB promise

Same audit. The heap alone was bigger than the request.

production-manifests.yaml+2 -2
removed:     ES_JAVA_OPTS: "-Xms4g -Xmx4g"
added:     ES_JAVA_OPTS: "-Xms2g -Xmx2g"
  resources:
    requests:
      memory: 3Gi
    limits:
removed:       memory: 6Gi
added:       memory: 3Gi

Elasticsearch needs memory that is not heap. Lucene leans on the filesystem cache, and the JVM has its own overhead outside -Xmx. A heap bigger than the whole request leaves none of that, and the gap only shows up under real search traffic.

2 GB of heap in a 3 GB pod leaves a third for everything else. Request equals limit again, for the same reason as Redis.

03 · Same cluster · backend · 12 pods

Node was allowed to grow to double what it asked for

Same audit, third one in a row. By now it was a pattern, not a typo.

production-manifests.yaml+1 -1
removed:     NODE_OPTIONS: "--max-old-space-size=3072"
added:     NODE_OPTIONS: "--max-old-space-size=1024"
  resources:
    requests:
      memory: 1500Mi
    limits:
      memory: 3Gi

V8 only collects hard when it gets close to its heap ceiling. With the ceiling at 3 GB the process grew well past the 1.5 GB it requested, so the nodes looked half empty to the scheduler while really being full.

At 1024 the garbage collector works earlier, the pod stays near what it asked for, and the extra limit is headroom for buffers instead of a second heap.

Three services, the same mistake. Nobody had ever lined up the process setting with the pod setting because they live in different parts of the file and usually different people's heads.

04 · Same cluster · four Services

I wrote this. It did nothing.

Spotted in the same manifests, during the move to an ALB.

production-manifests.yamldead config
kind: Service
metadata:
  name: frontend
  annotations:
dead line:     service.beta.kubernetes.io/aws-load-balancer-type: "alb"
spec:
dead line:   type: ClusterIP

Two reasons this is dead. A ClusterIP Service never gets a load balancer at all. And alb is not a value that annotation takes, it understands nlb and external.

Nothing errored. Kubernetes stores annotations it does not understand and carries on, which is exactly how a line like this survives for months looking important. It was on four Services.

An ALB comes from an Ingress with ingressClassName alb, picked up by the AWS Load Balancer Controller. That is the next receipt.

05 · Same cluster · frontend only

Moving the frontend off the NLB so browsers get HTTP/2

Every asset on the site was loading over HTTP/1.1. The app was fine. The load balancer was the wrong kind.

production-manifests.yaml+15 -0
added: apiVersion: networking.k8s.io/v1
added: kind: Ingress
added: metadata:
added:   name: frontend
added:   annotations:
added:     alb.ingress.kubernetes.io/scheme: internet-facing
added:     alb.ingress.kubernetes.io/target-type: ip
added:     alb.ingress.kubernetes.io/certificate-arn: <acm certificate>
added:     alb.ingress.kubernetes.io/ssl-redirect: "443"
added:     alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
added:     alb.ingress.kubernetes.io/load-balancer-attributes: |
added:       routing.http2.enabled=true,
added:       idle_timeout.timeout_seconds=300
added: spec:
added:   ingressClassName: alb

Before, ingress-nginx sat behind a network load balancer with the certificate in ACM. TLS ended on the NLB and nothing offered h2, so every browser fell back to HTTP/1.1 and queued its requests six at a time.

Now the frontend is behind an ALB created by the AWS Load Balancer Controller. target-type ip sends traffic straight to pod IPs, no extra hop through a node port.

HTTP/2 is on by default on an ALB. I set it anyway, so the next person reading this does not need to know that. The backend and admin stayed on their NLBs. They did not need it, and moving them was risk for nothing.

proof$ curl -sI --http2 https://<frontend> | head -1HTTP/2 200

06 · Same cluster · AWS Load Balancer Controller · IAM

The controller could not get credentials, and the error never said why

No ALB appeared, whatever the Ingress said.

fix-oidc-provider.sh+2 -0
"Principal": {
  "Federated": "arn:aws:iam::<account>:oidc-provider/<cluster-oidc>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
  "StringEquals": {
added:     "<cluster-oidc>:sub": "system:serviceaccount:kube-system:aws-load-balancer-controller",
added:     "<cluster-oidc>:aud": "sts.amazonaws.com"
  }
}

Three things were wrong at the same time. The cluster's OIDC provider had never been created in IAM. The trust policy did not match the controller's service account. And there were two controller installs from earlier attempts, fighting.

Each one on its own gives you the same line in the logs: not authorized to perform sts:AssumeRoleWithWebIdentity. So fixing one of them changes nothing you can see, and you start doubting the fix that was actually right.

Order that worked: delete every old controller install, create the OIDC provider, recreate the role with sub and aud pinned to the exact service account, then install the controller once.