论文标题
以加速优化的对称传送
Symmetry Teleportation for Accelerated Optimization
论文作者
论文摘要
现有的基于梯度的优化方法在本地更新参数，以最大程度地减少损失函数的方向。我们研究了一种不同的方法，即对称传送，该方法允许参数在损耗级别集中行驶很大距离，以提高随后步骤中的收敛速度。传送利用优化问题的损失格局中的对称性。我们为优化和多层神经网络中的测试功能提供了损失的群体动作，并证明了传送的必要条件以提高收敛速度。我们还表明，我们的算法与二阶方法密切相关。在实验上，我们表明传送范围提高了梯度下降和Adagrad的收敛速度，包括测试功能，多层回归和MNIST分类，包括测试功能。
Existing gradient-based optimization methods update parameters locally, in a direction that minimizes the loss function. We study a different approach, symmetry teleportation, that allows parameters to travel a large distance on the loss level set, in order to improve the convergence speed in subsequent steps. Teleportation exploits symmetries in the loss landscape of optimization problems. We derive loss-invariant group actions for test functions in optimization and multi-layer neural networks, and prove a necessary condition for teleportation to improve convergence rate. We also show that our algorithm is closely related to second order methods. Experimentally, we show that teleportation improves the convergence speed of gradient descent and AdaGrad for several optimization problems including test functions, multi-layer regressions, and MNIST classification.
