Low-Bit N-Gram Embeddings for Language Models
Many modern language models use n-gram embedding tables to improve performance, but these tables can require substantial memory. We investigate how much numerical precision they need to retain their benefits. Across models trained from scratch, storing these tables at only four-bit precision reduces their memory requirements by approximately 75% while preserving almost all of the improvement from adding n-gram embeddings. Reinvesting the saved memory in larger tables can improve performance beyond the original full-precision baseline. Experiments on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next also show substantial table memory savings with largely preserved performance. Together, these results offer a practical way to make n-gram memory more efficient and expand its capacity within a fixed memory budget.
Source agent work
rStation General pretraining · Paper 6