Post #1393901

2025-08-21 11:32 UTC

@dashdsrdash @mhoye slightly aside from your main point here, I quite like that as a definition of "big data". If it's big data then it doesn't fit on a single computer and you need to do something distributed to get it to fit. Has some nice side benefits like accurately tracking that the definition of "big" changes every year as new components with larger storage capacity become available. Also "medium data" if it fits in SSDs on a single computer, "small data" if it fits in RAM (yes this can be multiple TBs ofc) and "tiny data" if it fits in RAM on a cheap computer.

Replies (3)

@AMS@infosec.exchange 2025-08-21 11:40

@0x2ba22e11 @dashdsrdash @mhoye I'd argue tiny data fits in the CPU cache (now 100MB). Small data fits in RAM/SSD (They're basically the same size: single digit TB). Medium data fits on one disk (tens of TB). Big data needs distributed storage and processing. Very few things are big or even medium data.

Open ##1393906
@shapr@recurse.social 2025-08-21 13:16

@0x2ba22e11 my laptop has two NVMe slots, and 8 TB drives are vaguely affordable. I have four TB drives but medium size sure has increased the past few years.

Open ##1393918
@mikebabcock@floss.social 2025-08-22 00:44

@0x2ba22e11 @dashdsrdash @mhoye I feel like there's a cross-product involved in both how much data there is and how many things you want to know about it. Indexing all the words in a library is not the same thing as doing textual analysis for tone and the average grade level of each chapter of every book with a green cover.

Open ##1393920