TLDR Tech / Hardware
Micron reportedly explores near-GPU NAND, a flash tier sitting between HBM and the SSD
Micron is reportedly investigating near-GPU NAND, high-endurance flash modules placed right on the GPU package where HBM sits today, aiming to fill the massive latency and bandwidth gap between HBM and a system's SSD. The pitch is memory-bound inference: large models buy extra accelerators purely to hold weights and KV cache, an expensive way to add capacity since HBM is the priciest, tightest-supplied memory around, so a cheaper flash tier fast enough to keep compute fed could let the same model run on fewer GPUs.
Read the full story