Post #2185958
2026-05-07 22:42 UTC
Replies (8)
-
@odr_k4tana@infosec.exchange 2026-05-07 22:56
@LeafyEricScott@hachyderm.io I think I know how, but it's too late for me today. I'll test this tomorrow and get back to you.
-
@chiraag@mastodon.online 2026-05-07 23:15
@LeafyEricScott@hachyderm.io I'm not sure about R (haven't used it in a _long_ time), but in Haskell I'd create a datatype for the row (Row Double String (Maybe Bool)) and create a parser that splits based on ',' (or whatever delimiter) and reads in each field. As might have been apparent, we allow for the third field in the data type to be `Nothing` or `Just `, and an Integer can be parsed as a Double, so no problem there either.
-
@nxskok@cupoftea.social 2026-05-07 23:19
@LeafyEricScott@hachyderm.io what works, at least in this case, is first to get the file names: fnames % bind_rows() This works here because bind_rows sorts out the column types (it makes column a "dbl") and if a column is missing in some files, it fills it with NAs. (bind_rows works with a list of dataframes such as is produced by this map.)
-
@rdnielsen@floss.social 2026-05-07 23:50
@LeafyEricScott@hachyderm.io Somewhat coincidentally, Daniel Lakeland recently asked a similar question, but he was using Julia. My recommendation is the same: import the data to a database using SQL tools and then connect to the database or export to a cleaned CSV. Here's a SQL tool that has all the metacommands and configuration options you need: https://pypi.org/project/execsql/ (disclaimer: I wrote it).
-
@pjacock@fediscience.org 2026-05-08 00:29
@LeafyEricScott@hachyderm.io @adamhsparks@rstats.me are these optional columns at the end of each line? Otherwise seems horribly heuristic.
-
@defuneste@fosstodon.org 2026-05-08 02:25
@LeafyEricScott@hachyderm.io You do not want (can't?) specify a column type while reading the multiple csv? You can always read as text and cast it later (worst case scenario). In the blind, I would use duckdb to read them with globing (defining the column type I expect) and specify what I do with error (see store_reject = true). That will not solve your "sometimes 3 columns or not". I would use {targets} with a custom function that check and "clean" every csv and then append.
-
@pkw@snac.d34d.net 2026-05-08 04:35
Have you considered pre-processing and normalizing the CSVs? (In a possibly more performant language/lib)
-
@geospacedman@mastodon.social 2026-05-08 08:49
@LeafyEricScott@hachyderm.io data table `fread` combined with rbind(..., fill=TRUE) does it, I think, and for your benchmark its 5-10x faster, but obvs this needs scaling to real world data sizes. [in bench:mark] dt = do.call(rbind, lapply(csvs,fread)), dt_fill= do.call(\(...)rbind(...,fill=TRUE), lapply(csvs,fread)), The `dt` line works for the first 2 CSVs, `dt_fill` works for all three. Here's comparative output for the 2 CSV case.