01 · Consumer travel platform · EKS · 6 Redis pods
Redis was told it could have more memory than it asked for
Found by lining up every memory setting in the cluster against the container it runs in.
env: removed: REDIS_ARGS: "--appendonly yes --maxmemory 3gb --maxmemory-policy allkeys-lru" added: REDIS_ARGS: "--appendonly yes --maxmemory 1536mb --maxmemory-policy allkeys-lru" resources: requests: memory: 2Gi limits: removed: memory: 4Gi added: memory: 2Gi
The scheduler packs nodes by requests, not limits. Each Redis pod asked for 2 GB and was then allowed to fill 3 GB of cache, plus whatever the AOF rewrite fork copies on top. Six of those on a busy node, and the memory everybody was counting on is not there. The kernel picks a victim and it is usually the biggest one.
Now maxmemory sits 512 MB under the limit, and request equals limit, so what the scheduler adds up is what actually gets used.
$ kubectl top pods -l app=redisevery pod under 2Gi, with room for the fork