Post #1393906
2025-08-21 11:40 UTC
Replies (5)
-
@dashdsrdash@tilde.zone 2025-08-21 12:17
@AMS @0x2ba22e11 @mhoye These are all pretty reasonable definitions, but I think the fundamental distinction is just: we need more than one commodity server to store or process the data fast enough for our purposes vs one commodity server will do That is to say, do we have a coordination/distribution/parallelism problem to solve, or not? I specify a generic commodity server so that we don't have to think about exotica like Oxide or IBM z or whatever OCP-adjacent system the FAAMNGs are using: those are all solutions on the big side. If you can configure and buy it without talking to a salescritter, have it arrive within 30 days and then put it on a desk or in a rack, it's a commodity server, close enough.
-
@dalias@hachyderm.io 2025-08-21 23:32
@AMS @0x2ba22e11 @dashdsrdash @mhoye Big data is at the scale of "full text index of the internet". Anything else is not big data.
-
@mikebabcock@floss.social 2025-08-22 00:39
@AMS as someone who does hobby #photography with a #DSLR my average day out shooting is about 30-50GB and it still only takes a few minutes to fully index that content in both #darktable (for editing) and #immich (for search). The *entirety* of my last 12+ years of shooting is a much larger dataset, but its also static. The whole concept of what is "big" is hard to fathom these days.
-
@log@mastodon.sdf.org 2025-08-22 14:56
@AMS @0x2ba22e11 @dashdsrdash @mhoye I might even argue that "big data" must still be "big" even after removing data that is only being used to search for significance or train associative or probabilistic models, and probably never should have been collected at all, much less stored long-term.
-
@jima@mspsocial.net 2025-08-26 14:52
@AMS @0x2ba22e11 @dashdsrdash @mhoye I wouldn't assume RAM is single-digit TB. 👀