Post #2280773

2025-10-30 11:58 UTC

@sminez@hachyderm.io i think you can do the same with streaming, _if_ you know all the matching which needs to happen on a chunk (or up to an offset) has happened? Hence API above. I get that the parallel bit was hard, but it sounds like you have got that working for non-streaming case (well done :-) ) I think I mean...run your first pass on the stream and periodically ask your matchers for the offset they are finished with. You can then run your second pass on the stream up to that point, emit and discard.

Replies (1)

@sminez@hachyderm.io 2025-10-30 13:09

@jbert@hachyderm.io yeah the "if you know all the matching which needs to happen on a chunk" part is what I'm going back and forth on but the more I think about it the more I'm suspecting that I'm making things more complicated than they need to be. My worry is that it would be possible to write a group that produced match positions out of order once you account for extending the current "dot" forward and backward (covered in the paper). That said, I've not actually written such a structex so I need to check to see if that's even possible. If it's not then the design you're outlining should work I think? I have some thoughts on an implementation I'd like to try but I want to make sure I'm not just digging a hole for myself before I try it out(!)

Open ##2280774