Post #4487100

2026-08-09 15:15 UTC

Here's the first puzzle, a warm-up. This is a recent heat map of memory usage in an HTTP service. What caused these four prominent "spurs" you see along the top? (More context, not all necessarily relevant: The service is running in Kubernetes, so the samples represent containers, each container running multiple workers. The Y axis, not shown, starts at zero. Time zone for X axis is unspecified.) Anything else can you figure out from this graph? #GraphMystery

Replies (1)

@c0coChannel@infosec.exchange 2026-08-10 11:21

the timing on this is interesting, assuming the X axis dates are at 00:00:00, the spurs appear late eve and ramp up in the early hours before dropping off completely, with substantially reduced memory usage in those mid ranges during those periods too trying to reason about what the Y axis actually is (since it's a heatmap, assuming the color is memory usage). Potentially Lifetime/uptime/duration of clusters/workers. There would be an average range for lifetimes (The main heat strip through the middle of the graph where most points sit), and the spurs would be those anomaly clusters/workers that have kept running for much longer than average whereas others are resetting at some given breakpoint, until ultimately the spurs reach a fixed maximum lifespan and are killed by orchestration. A guess for why this would happen is that these workers are managing redis cache and / or compression/backups, meaning they keep running longer than the bulk of the clusters which are handling the HTTP traffic. (continuing to assume the Y axis is lifetimes) The heat map would be showing highest memory usage at the peak of the average lifetime generally, with two main ramp-ups in memory usage across all during a couple of points throughout the day - peak traffic times affecting all workers. Am I close, or wildy and embarassingly wrong?

Open ##4487099