TLDR Tech / Practical
Byte-Level Language Models Achieved With New Retrofitting Method
A new general approach called byteification allows existing large language models that use subwords to be adapted for byte-level operations with minimal additional training. By introducing a non-causal boundary predictor and a two-stage training process to address a mismatch in patch boundary prediction, the resulting models achieve performance comparable to state-of-the-art subword models. They also demonstrate superior capabilities in character-level reasoning and scientific tasks. This method further enables byte-level models to increase inference speed and utilize existing post-trained components through task arithmetic.
Read the full story