Post #2312779

Replies (1)

• @nxskok@cupoftea.social 2026-05-08 00:39

  @LeafyEricScott@hachyderm.io ok then, some random thoughts: - I doubt that a list containing all the dataframes is going to be much bigger memory-wise than your final combined dataframe (that you presumably want to have in memory anyway). - You could go absolutely nuts in trading time for memory something like this: Write a function f1 that takes a csv filename and returns a vector of column names (by reading in the csv file and running names(), say) Run f1 on all of your files one by one. The distinct column names you get are all the column names you are looking out for. Write a function f2 that takes a csv filename and returns a regularized dataframe: that is, one that has columns with *all* the names you expect, with (say) missing values for any columns that weren't in the original. Run f2 on all your files one by one, saving the result in (say) an rds file each time. Then read in your rds files one by one and assemble your final dataframe.

  Open ##2312780