论文标题
有条件的随机优化和元学习中应用的偏见随机一阶方法
Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learning
论文作者
论文摘要
条件随机优化涵盖了从不变学习和因果推断到元学习的各种应用。但是，由于组成结构，为此类问题构建无偏梯度估计器是具有挑战性的。作为替代方案，我们提出了一个有偏见的随机梯度下降（BSGD）算法，并研究了不同结构假设下的偏见变化权衡。我们在光滑且非平滑的条件下建立了BSGD的样品复杂性，用于强烈凸，凸和弱凸目标。我们的下限分析表明，对于一般凸目标和非凸目标，BSGD的样本复杂性无法提高，除了具有Lipschitz连续梯度估计器的平滑非凸目标外。对于这种特殊设置，我们提出了一种称为偏置蜘蛛杆（Bspiderboost）的加速算法，该算法与下界复杂性匹配。我们进一步对不变的逻辑回归和模型 - 不合稳定元学习进行数值实验，以说明BSGD和BSPIDERBOOST的性能。
Conditional stochastic optimization covers a variety of applications ranging from invariant learning and causal inference to meta-learning. However, constructing unbiased gradient estimators for such problems is challenging due to the composition structure. As an alternative, we propose a biased stochastic gradient descent (BSGD) algorithm and study the bias-variance tradeoff under different structural assumptions. We establish the sample complexities of BSGD for strongly convex, convex, and weakly convex objectives under smooth and non-smooth conditions. Our lower bound analysis shows that the sample complexities of BSGD cannot be improved for general convex objectives and nonconvex objectives except for smooth nonconvex objectives with Lipschitz continuous gradient estimator. For this special setting, we propose an accelerated algorithm called biased SpiderBoost (BSpiderBoost) that matches the lower bound complexity. We further conduct numerical experiments on invariant logistic regression and model-agnostic meta-learning to illustrate the performance of BSGD and BSpiderBoost.
