TLDR Tech / Data
oLLM (GitHub Repo)
oLLM is a Python library for running very large context LLM workloads on modest consumer hardware by offloading model weights and KV cache to SSD instead of keeping everything in GPU memory. It's useful for offline tasks like analyzing long documents, logs, contracts, chats, or reports locally without quantization.
Read the full story