论文标题
通过降低最大编码率的原理学习多样化和判别性表示
Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction
论文作者
论文摘要
为了从最大程度地区分类别的高维数据中学习内在的低维结构，我们提出了最大编码率降低的原理（$ \ text {mcr}^2 $），这是一种信息理论措施，该措施使整个数据集和每个类别的总和之间的编码率差异最大化。我们阐明了它与大多数现有框架的关系，例如跨凝结，信息瓶颈，信息增益，承包和对比度学习，并为学习多样化和歧视性特征提供了理论保证。可以准确地从类似于子空间的分布的有限样本中准确计算编码率，并可以以统一的方式学习监督，自我监督和无监督的设置中的内在表示。从经验上讲，仅使用此原理学到的表示形式比使用跨透明拷贝的表示形式在分类中标记损坏的标签要高得多，并且可以导致最先进的结果导致从自我学习的不变特征中聚集混合数据。
To learn intrinsic low-dimensional structures from high-dimensional data that most discriminate between classes, we propose the principle of Maximal Coding Rate Reduction ($\text{MCR}^2$), an information-theoretic measure that maximizes the coding rate difference between the whole dataset and the sum of each individual class. We clarify its relationships with most existing frameworks such as cross-entropy, information bottleneck, information gain, contractive and contrastive learning, and provide theoretical guarantees for learning diverse and discriminative features. The coding rate can be accurately computed from finite samples of degenerate subspace-like distributions and can learn intrinsic representations in supervised, self-supervised, and unsupervised settings in a unified manner. Empirically, the representations learned using this principle alone are significantly more robust to label corruptions in classification than those using cross-entropy, and can lead to state-of-the-art results in clustering mixed data from self-learned invariant features.
