Post #3222295
2026-01-07 14:58 UTC
@osma@sigmoid.social sorry for the belated response!
1) we only used completion, i.e. output, tokens as that is what the tools we used do, too, although I’m not 100% sure how this is justified.
Replies (1)
-
@osma@sigmoid.social 2026-01-07 15:17
@sarahalang@fedihum.org Thanks for confirming! I think that prompt (input) processing is important too, especially for tasks where the input size is large relative to the output (e.g. in document classification, where the input can be long and the output is just a few tokens). PP is typically much faster than generation, but can still be a significant factor in response time and thus energy use.