TLDR Tech / AI
Emergent Modularity in Mixture-of-Experts Models
Allen AI released EMO, a mixture-of-experts model that learns modular expert organization directly from pretraining data instead of predefined domains. Tasks can run on just 12.5% of the experts while preserving near full-model performance.
Read the full story