Post #3532637

2026-07-01 17:26 UTC

@partim@social.tchncs.de If we’re a few people who can spend some time, we should be able to create something that works well. My choice would also to build this in Rust (although it’s new to me). I was thinking something like a hybrid SAX parser that creates a stream of individually self-contained objects. Maybe there are better ideas. I’ll check what process we should have for new people to join at this time, I’ll get back with a DM!

Replies (1)

@partim@social.tchncs.de 2026-07-01 17:41

@stefanlindbohm@mastodon.social I’ve pushed the current state of my experiments here: https://codeberg.org/partim/transdata There isn’t an awful lot, it is quite messy and woefully under-documented. But the idea is indeed a SAX-style parser that produces a stream of data objects. Things are complicated a bit by a very Rust-y ambition to allocate as little as possible, which is probably reasonable, given that the Swiss dataset is 22 Gigs.

Open ##3532636