Post #3222296
2026-01-07 15:17 UTC
@sarahalang@fedihum.org Thanks for confirming! I think that prompt (input) processing is important too, especially for tasks where the input size is large relative to the output (e.g. in document classification, where the input can be long and the output is just a few tokens). PP is typically much faster than generation, but can still be a significant factor in response time and thus energy use.
Replies (2)
-
@osma@sigmoid.social 2026-01-07 15:22
@sarahalang@fedihum.org It seems to me that your LLM usage was quite prompt-heavy since the documents were quite long (IIRC you didn't state how many tokens average/median) and the output was just a classification (relevant/irrelevant) plus a few bits of structured information. Thus your energy use was likely severely underestimated. 40 kWh is not that much.
-
@sarahalang@fedihum.org 2026-01-07 15:23
@osma@sigmoid.social that’s honestly also something I had worried about.. ultimately we’d need something for the #DH that takes into account both inputs and outputs…