Post #3532636
2026-07-01 17:41 UTC
@stefanlindbohm@mastodon.social I’ve pushed the current state of my experiments here: https://codeberg.org/partim/transdata
There isn’t an awful lot, it is quite messy and woefully under-documented. But the idea is indeed a SAX-style parser that produces a stream of data objects.
Things are complicated a bit by a very Rust-y ambition to allocate as little as possible, which is probably reasonable, given that the Swiss dataset is 22 Gigs.
Replies (1)
-
@stefanlindbohm@mastodon.social 2026-07-01 21:02
@partim@social.tchncs.de Oh, nice! I think efficiency is much needed for the 22 GB datasets, as you say. Imagine importing 30-40 countries once we get open data everywhere. So probably worth it to accept some complicated stuff for the sake of performance. Did you see my writeup on NeTEx the other day btw? Sounds like we both did the same thing of just jumping in and realising how bad things are 😅 https://gist.github.com/stefanlindbohm/58d11afbb9af2d5ec88263f62322caaa