TLDR Tech / Data
14x faster embeddings: how we rebuilt the ONNX path in Manticore
Manticore rewrote its embedding pipeline on ONNX Runtime, slashing CPU waste and lifting throughput up to 14x for low-latency vector search. The design shares one thread-safe ONNX session, disables intra-op spinning, and processes documents individually to avoid lock contention and variable-length padding overhead.
Read the full story