Post #918759

2026-03-20 13:10 UTC

Hey, #digipres folks, what algorithms do you use for periodic fixity checks on data at rest (esp. if stored in the cloud)? CRC32 is tempting, because it&#39;s much faster than MD5 (and thus presumably cheaper in terms of compute, emissions, and money). However, I get the impression that it&#39;s not much used for this purpose. Is that because of collision worries? Not being cryptographically secure? Something else?

Replies (2)

@p3ter@mastodon.social 2026-03-22 19:47

@linguistory@code4lib.social Unless you're actually handling sensitive data, which would attract hashcode-matching forged data, I'm not worried about cryptographic-requirements on the hash. For integrity checking. According to Reto K., mac seems to have hardware support for SHA - outspeeding MD5. Not so on PCs. On my notebook, MD5 is faster than CRC32, btw. (0,08 vs 0,136s)

Open ##2191121
@p3ter@mastodon.social 2026-03-22 19:49

@linguistory@code4lib.social periodicity of checks of data "on shelves" (#LTO?): Good question! Professionally, I'd say "it depends": Find a sweetspot which doesn't disrupt daily work, and a method which has a reasonable runtime (days regular, weeks is normal, months uncommon) Are you talking about data on shelves, or on a "nearline" backup copy on servers/HDDs? With LTO there's also the issue with Generations and drives...

Open ##2191122