论文标题
增强具有文本结构知识的预先训练的模型，以产生问题
Enhancing Pre-trained Models with Text Structure Knowledge for Question Generation
论文作者
论文摘要
如今，预先训练的语言模型对于问题产生（QG）任务取得了巨大的成功，并明显超过传统的顺序到序列方法。但是，预训练的模型将输入段视为平坦序列，因此不知道输入段的文本结构。对于QG任务，我们将文本结构建模为答案位置和句法依赖性，并提出答案局部性建模和句法掩盖的注意，以解决这些局限性。特别是，我们以高斯偏见为局部建模，以使模型能够专注于答案的上下文，并提出一种掩盖注意机制，以使输入段落的句法结构在问题生成过程中访问。在小队数据集上进行的实验表明，我们提出的两个模块可以改善强大的预训练模型ProPHETNET的性能，并将它们梳理在一起，可以通过最先进的预训练模型来实现非常有竞争力的结果。
Today the pre-trained language models achieve great success for question generation (QG) task and significantly outperform traditional sequence-to-sequence approaches. However, the pre-trained models treat the input passage as a flat sequence and are thus not aware of the text structure of input passage. For QG task, we model text structure as answer position and syntactic dependency, and propose answer localness modeling and syntactic mask attention to address these limitations. Specially, we present localness modeling with a Gaussian bias to enable the model to focus on answer-surrounded context, and propose a mask attention mechanism to make the syntactic structure of input passage accessible in question generation process. Experiments on SQuAD dataset show that our proposed two modules improve performance over the strong pre-trained model ProphetNet, and combing them together achieves very competitive results with the state-of-the-art pre-trained model.
