Post #4252708

At some point I guess I should name the ST:TNG voice interface project. "Holly" would be a bit too far fetched I guess. So far it has just been something I cobbled together where the originating speech-to-speech pipeline, with my additions, ran on my Nvidia 5060Ti 16GB workstation using an LLM that I always have running on another server, on an Nvidia A2000 12GB GPU. To make this actually usable it would need to run completely on that server, freeing up my workstation GPU for the important cybersec and dev tasks it's otherwise used for, so I've now spent the evening seeing just how far down I could bring the resource requirements of the voice pipeline. Success! The Parakeet STT was a 2GB model that was way overpowered for the needs of this project. Replaced with faster-whisper tiny.en at ~150MB. Qwen3-TTS originally used 3.6GB at FP16 weights, but the Q8_0 quantization coming in at 2.4GB sounds just as good to my ears. That, coupled with lowering the max context size of the Gemma4 12B QAT LLM meant I now have it all fully loaded on the A2000 GPU and can continue wiring up this Star Trek clone of a voice assistant throughout the house. I'm now quite far ahead in functionality from the original project so it will soon be time to decouple completely and simply the architecture a bit. This turned out to be a really fun little vacation activity :)