Context Is Software, Weights Are Hardware

2026-04-23

While context acts as temporary "software" for in-context learning, weight updates permanently modify a model's fundamental computational capabilities and internal representations. Effective continual learning requires both increasing context windows for working memory and persistent weight-space learning for accumulated, fundamental knowledge.