This feels like using a harrier jet to take in your groceries. Seems like something that could be built using one of the very small edge models - a 0.3 or 0.5b maybe with some fine tuning, and run on cpu on device for free without any real quality loss.

It definitely can, it just depends on the complexity of the commands you want it to generate. Most of the time a small local model will do what you want. If you know of a small local model fine tuned to generate shell commands I'm all ears. I usually run it with a 9B qwen model.

Pairing this with dictation is fun though, and speeds things up a lot for terminal heavy work.