#parquet

Jörn Franke @jornfranke@social.anoxinon.de

· 1mo ago

published scrapy-contrib-bigexporter 1.2.0 export scraped data using Scrapy into big data formats, such as Iceberg, Parquet, Orc, Avro Changelog: https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexporters/src/branch/main/CHANGELOG.md #python #scrapy #parquet #iceberg #scraping #web

codeberg.org

Cookie monster!

Lobsters @lobsters@robot.villas

· 2mo ago

A Fast Path for Fixed-Length Lists in Parquet https://www.morling.dev/blog/fast-path-for-fixed-length-lists-in-parquet/ #DataEngineering #Performance #Parquet

Gunnar Morling

A Fast Path for Fixed-Length Lists in Parquet

In its current form Apache Parquet isn’t a great fit for storing fixed-length lists, such as coordinates, RGB(A) colors, or—an increasingly common case—vector embeddings driving search and …

You've seen all posts