Table of contents
background
When human beings observe external things or read articles, such as viewing a picture, or viewing a web page, or reading an article, the attention of human eyes will be more inclined to observe or read some important local information , and integrate the local information of different regions, so as to quickly establish an overall overview of the observed thing or the read article. So the attention mechanism is to give different weights to the local information of the data such as pictures and texts to be processed, so as to achieve a certain task. So attention is a data processing method, which can be understood as the weight of local information.
Attention Mechanism was first applied in the image field, and the idea was proposed in 1990s.
The Attention mechanism was first applied in NLP tasks, mainly to optimize and improve the importance of local information to the prediction results, so that the encoder and decoder can learn higher sequence information.
An article about the attention mechanism from Google in 2017:
Attention is all you need https://arxiv.org/pdf/1706.03762.pdf
continue writing tomorrow