I might have said this earlier but worth repeating. DuckDB is freaking amazing! Today I worked on a 56G TSV file containing 55 million rows of JSON serialized documents (all of OpenLibrary editions). I can query this file including filtering on nested JSON structures and most queries complete in a few dozen seconds worst case, but I can also create derived indexes to bring down query times to milliseconds. All in memory on an M1 Mac with 64GB RAM. #DuckDB #OpenLibrary #BigData