TLDR Tech / Practical
Laptop CPU powers Vietnamese restaurant receptionist with LLM last
An engineering project successfully created a local Vietnamese restaurant phone receptionist that operates entirely on a laptop's CPU, requiring no GPU. By prioritizing deterministic code and skills before engaging a large language model, and using prompt-caching techniques with llama.cpp, the system achieves rapid response times, ranging from 0.02 to 0.30 seconds. The design employs deterministic code for business functions like reservations and pricing, while optimizing LLM performance through prompt prefilling and raw mode templating.
Read the full story