Post #2312798
2026-05-08 08:49 UTC
@LeafyEricScott@hachyderm.io data table `fread` combined with rbind(..., fill=TRUE) does it, I think, and for your benchmark its 5-10x faster, but obvs this needs scaling to real world data sizes.
[in bench:mark]
dt = do.call(rbind, lapply(csvs,fread)),
dt_fill= do.call(\(...)rbind(...,fill=TRUE), lapply(csvs,fread)),
The `dt` line works for the first 2 CSVs, `dt_fill` works for all three. Here's comparative output for the 2 CSV case.
Replies (3)
-
@danwwilson@rstats.me 2026-05-08 09:31
@geospacedman@mastodon.social @LeafyEricScott@hachyderm.io I was going to suggest something like this too and I’d import col’s as characters and cast columns once imported.
-
@tim_salabim@fosstodon.org 2026-05-08 12:36
@geospacedman@mastodon.social @LeafyEricScott@hachyderm.io throw some data.table::rbindlist into that mix and you should be good to go. It actually has an example on its help page that is rather similar to the OPs question. If needed it will also let you add a column indicating the source (file) of the rows.
-
@LeafyEricScott@hachyderm.io 2026-05-08 17:14
@geospacedman@mastodon.social Looks like the memory usage for the fread methods is much higher, so that's not going to help