Post #1645028
2026-04-23 07:56 UTC
@jimbob having dealt with computations on large multidimensional arrays, a solution I found was to use the HDF5 format. This supports massive arrays of several data types (with optional compression), which can be lazy-loaded and lazy-written, so you can handle things slice-by-slice or line-by-line without overloading your memory. Be careful with selecting the right chunk size and dimensions optimized to your problem.
Replies (1)
-
@jimbob@aus.social 2026-04-23 07:58
@doc yes I think that might be my next step, to be honest. The input data came as NetCDF so I was attempting to stick with it so the additional variables I'm calculating are in the same format, but honestly I might just convert everything over.