论文标题
用各向同性迭代量化嵌入压缩
Embedding Compression with Isotropic Iterative Quantization
论文作者
论文摘要
单词的连续表示是基于深度学习的NLP模型中的标准组件。但是，代表大型词汇需要大量的内存，这可能会引起问题，尤其是在资源约束平台上。因此，在本文中，我们提出了一种将嵌入向量压缩到二元的方法的各向同性迭代量化方法（IIQ）方法，利用迭代量化技术为图像检索良好，同时满足基于PMI的所需的各向同性特性。具有预训练的嵌入（即手套和HDC）的实验表明，与原始的实数嵌入载体相比，相当甚至改善的性能相当甚至改善了31倍以上。
Continuous representation of words is a standard component in deep learning-based NLP models. However, representing a large vocabulary requires significant memory, which can cause problems, particularly on resource-constrained platforms. Therefore, in this paper we propose an isotropic iterative quantization (IIQ) approach for compressing embedding vectors into binary ones, leveraging the iterative quantization technique well established for image retrieval, while satisfying the desired isotropic property of PMI based models. Experiments with pre-trained embeddings (i.e., GloVe and HDC) demonstrate a more than thirty-fold compression ratio with comparable and sometimes even improved performance over the original real-valued embedding vectors.
