【目的】總結注意力機制在自然語言處理領域的衍化及應用規律。【文獻范圍】以"attention"和"注意力"為檢索詞,分別檢索WoS、The ACM Digital Library、arXiv以及中國知網,時間跨度限定為2015年1月至2019年10月,制定標準人工篩選自然語言處理領域的文獻,最終獲得68篇相關文獻。【方法】在深入分析文獻的基礎上,歸納注意力機制的通用形式,梳理其衍生類型,并基于數據對其在自然語言處理任務中的應用情況進行述評。【結果】注意力機制在自然語言處理中的應用集中于序列標注、文本分類、推理以及生成式任務,且任務和注意力機制的類型之間存在一定的適配規律。【局限】部分注意力機制和任務間的適配結論是通過模型整體表現數據間接得出的,不同注意力機制間的性能差異有待進一步研究。【結論】注意力機制的研究切實推進了自然語言處理的發展,但其作用機理尚未明了,提高其可解釋性并使之更加接近人類的真實注意力是未來的研究方向。

[Objective] This paper summarizes the evolution and application of attention mechanism in natural language processing. [Coverage] We searched"attention"with the title/topic fields of WoS, ACM Digital Library,arXiv and CNKI from January 2015 to October 2019. Then, we manually screened the topic literature in the field of natural language processing, and obtained 68 related papers. [Methods] We first summarized the general attention mechanism, and sorted out its derivations. Second, we thoroughly reviewed their applications in natural language processing tasks. [Results] The application of attention mechanism in natural language processing focused on sequence labeling, text classification, reasoning and generative tasks. There were adaptation rules between tasks and the various attention mechanisms. [Limitations] Some adaptations between the mechanisms and the tasks were obtained from the overall performance of the model. More research is needed to examine the performance of different mechanisms. [Conclusions] The study of attention mechanism has effectively promoted the development of natural language processing. However, the mechanism of action is not yet clear. Future research should focus on making attention mechanism closer to those of the human beings.