Post #2312804
2026-05-08 18:04 UTC
@LeafyEricScott@hachyderm.io Oh dear. How much memory is too much memory? Maybe reading one-at-a-time and rbinding each - in a for-loop - would prob be slower but less memory
Replies (1)
-
@LeafyEricScott@hachyderm.io 2026-05-08 19:22
@geospacedman@mastodon.social For this particular project, I think I need a database-like approach. I had high hopes for rearranging CSVs in hive style partitioning and using `arrow::open_dataset()`. Looks like DuckDB is going to be the way to go. But in general, it would be nice for `read_csv(files, col_select = 1:5)` to "just work" even when some of the files have 6 columns, because it's a lot more performant than `map(files, read_csv) |> list_rbind()` (or the data.table equivalent, I'm guessing)