Post #2312791
2026-05-08 02:25 UTC
@LeafyEricScott@hachyderm.io You do not want (can't?) specify a column type while reading the multiple csv? You can always read as text and cast it later (worst case scenario).
In the blind, I would use duckdb to read them with globing (defining the column type I expect) and specify what I do with error (see store_reject = true).
That will not solve your "sometimes 3 columns or not". I would use {targets} with a custom function that check and "clean" every csv and then append.
Replies (3)
-
@defuneste@fosstodon.org 2026-05-08 02:26
@LeafyEricScott@hachyderm.io if you do not need column 3, I would `cut` in shell it before reading the csv.
-
@LeafyEricScott@hachyderm.io 2026-05-08 02:50
@defuneste@fosstodon.org I want all the columns! I can figure out all possible columns and their types, so I could add missing columns to all the CSVs (or convert them to parquet, because I might as well if I'm doing this).
-
@LeafyEricScott@hachyderm.io 2026-05-08 02:51
@defuneste@fosstodon.org and if I could just specify columns and types I would! That's what I try to do multiple ways in the blog post. Sure seems like it should be possible!