Post #1049044 · Elektrine

2026-04-04 08:41 UTC

I am seeing up to 30% wall-time fluctuation running the same code on the same data on the same VM type in GCE. That seems crazy to me. What&#39;s the worst variation you&#39;ve seen? How do you deal with these things? I mean, for any benchmarking you need to run the clickhouse tomato benchmark protocol? (E.g. run two instances of the same software on the same VM so they&#39;re exposed to the same noisy neighbor effects).

Replies (3)

@wirepair@mastodon.social 2026-04-04 08:51

@HalvarFlake@mastodon.social oh boy don't get me started, I had to build a benchmarking system for GitLab and we had to buy sole tenants, the variablity of noisey neighbors is just too high to do any realistic benchmarking.

Open ##2620954
@Aissen@social.treehouse.systems 2026-04-04 09:52

@HalvarFlake@mastodon.social very in line with what we measured, heavily follows rush hour per geographic area. Yet another advantage for on-prem.

Open ##2620955
@slowcoding@toot.community 2026-04-04 14:07

@HalvarFlake@mastodon.social even measuring latency in a busy VM is not reliable, as usually time drift is amortized by hypervisors as we found out 10 years ago with our benchmark team so wall-clock lies in VMS... If you need the 99p bare metal is your only real option. But do you really need it? Most want it (as we promised our clients) but don't really need it(as something fails hard) though

Open ##2620957