Introduction to self-attention and code handwriting

self-Attention

---

Self-Attention Architecture

The way self-attention works is to input a row of vectors and output a row of vectors.

The output vector takes into account the information of all the input vectors.

• Self-attention can be superimposed many times.
  
  Fully connected layers (FC) and Self-attention can be used interchangeably.

1. Self-attention processes the information of the entire Sequence
2. FC's Network, focusing on handling inquiries from a certain location

The process of Self-Attention

• One of its architectures is like this, the output is a row of $b$ is$a$ is calculated and output.

• $b^1$ is considered$a^1,a^2,a^3,a^{}$generated after${}^{4\; .}$

• $b^2,b^3,b^4$ also consider$a^1,a^2,a^3,a^4$ , their calculation principles are the same.

  Computes the correlation of two input vectors

• Two common calculation methods: dot product and Additive

• Among them, in the first method, first let these two vectors be multiplied by a W (these two w are different, one is weight_q, the other is weight_k), respectively get q, kq, $q,k$ , and then do the inner product to get a score$a_{i,j}$Indicates the calculation of $a_{i}and{a}_j$of relevance.

• After $a^1$ minute sum$a^2,a^3,a^4$ Calculate the similarity.$a^1$ target$q^1$ minute sum$a^2,a^3,a^{}$k 2 , k 3 , k 4 of 4 k^{2},k^{3},k${}^{^}$ {4}$k^2,k^3,k^4$ Do the inner product to get attention scorce$a_{1,2}$,$a_{1,3}$,$a_{1,4}$, a 1 a^{1} will also be calculated in actual operation$a^1$ similarity to yourself.

• Subsequent future $a_{1,1},a_{1,2}$,$a_{1,3}$,$a_{1,4}$, get a 1 , 1 ′ a^{'}_{1,1} after a soft-max layer$a_{1,1}^{{}^{\prime }}$ $a_{1,2}^{{}^{\prime }}$ …
  

• Then put $a^1$ to$a^{}$Each vector of${}^4$$W^v$ gets a new vector, respectively get$v^1,v^2,v^3,v^4$

• Next, $v^1$ to$v^4$ , each vector is multiplied by the Attention score, and then added up to get$b^1$
  $$b^1=\sum _i{a}_{1,i}^{{}^{\prime }}{v}^i$$

• Theory, $a^2,a^3,a^4$ also do the same operation to get$b^2,b^3,b^4$ .

matrix angle

Each input vector will generate a set of $q^i,k^i,v^i$

Therefore, we will each $a^i$ left multiplied by a$w^q$ matrix, you can get the Q matrix by doing matrix multiplication, and each column of the Q matrix is ​​each input$a^i$的$q^i$.

Similarly each group $k^i,v^i$ can also be calculated with a matrix.

We know that each attention$a_{i,j}$Represents qiq^{i} of the i-th input$q^{}$kjk^{j}${}^{of\; i}$ and the jth input$k^j$ inner product.

Then, these four steps can be obtained by multiplying the matrix and the vector.

Furthermore, we can calculate all the attention and get$A^{{}^{\prime }}$

while$b^1$ is by using$v^i$ left side$A^{{}^{\text{'}}}$ get,

review

​ Figure (15)

• I is the input of self-attention, and the input of self-attention is a row of vectors, which are put together as columns of the matrix.
• Input is multiplied by three matrices $$w^q,w^k,w^v$$ gets Q, K, V.
• Next, Q is multiplied by the transpose of K to obtain matrix A, and after softmax processing, $A^{{}^{\prime }}$ , and then left multiplied by V to get Output.
• Therefore, the only parameter to learn in self-attention is the W matrix, which is the part that requires network train.

---

code example

The example is divided into the following steps:

1. ready to enter
2. Initialize weights
3. Export representation of key, query and value
4. Calculating attention scores
5. Calculate softmax
6. Multiply attention scores by value
7. Sum the weighted values ​​to get the output

Assuming input:

    Input 1: [1, 0, 1, 0]     
    Input 2: [0, 2, 0, 2]  
    Input 3: [1, 1, 1, 1]

Initialization parameters

Since our input is three 4-dimensional vectors, the figure (15) needs to be multiplied by W from the left, and the dimension of W is set to (4,3),

$$w^k,w^q,w^v$$ are w_key, w_query, w_value respectively.

   x = [
    [1,0,1,0], # 输入1
    [0,2,0,2], #输入2
    [1,1,1,1], #输入3
    ]
    x = torch.tensor(x,dtype=torch.float32)
    # 初始化权重
    w_key = [
        [0, 0, 1],
        [1, 1, 0],
        [0, 1, 0],
        [1, 1, 0]
        ]
    w_query = [
        [1, 0, 1],
        [1, 0, 0],
        [0, 0, 1],
        [0, 1, 1]
        ]
    w_value = [
        [0, 2, 0],
        [0, 3, 0],
        [1, 0, 3],
        [1, 1, 0]
        ]
    # 转化成tensor数据类型
    w_key = torch.tensor(w_key,dtype=torch.float32)
    w_query = torch.tensor(w_query, dtype=torch.float32)
    w_value = torch.tensor(w_value, dtype=torch.float32)

View QKV

querys = torch.tensor(np.dot(x,w_query),dtype=torch.float32)

keys = torch.tensor(np.dot(x,w_key) ,dtype=torch.float32)

calculate attention

# get attention scorce
attention_scores = torch.tensor(np.dot(querys,keys.T))

softmax processing

# 计算soft-max
attention_scores_softmax = torch.tensor( softmax(attention_scores,dim=-1) )

Multiply attention scores by value

weight_values = values[:,None] * attention_scores_softmax.T[:,:,None]

Sum the weighted values ​​to get the output

outputs = weight_values.sum(dim=0)

test code

import torch
import numpy as np
from torch.nn.functional import softmax
def preData():
    #
    x = [
    [1,0,1,0], # 输入1
    [0,2,0,2], #输入2
    [1,1,1,1], #输入3
    ]
    x = torch.tensor(x,dtype=torch.float32)
    # 初始化权重
    w_key = [
        [0, 0, 1],
        [1, 1, 0],
        [0, 1, 0],
        [1, 1, 0]
        ]
    w_query = [
        [1, 0, 1],
        [1, 0, 0],
        [0, 0, 1],
        [0, 1, 1]
        ]
    w_value = [
        [0, 2, 0],
        [0, 3, 0],
        [1, 0, 3],
        [1, 1, 0]
        ]
    # 转化成tensor数据类型
    w_key = torch.tensor(w_key,dtype=torch.float32)
    w_query = torch.tensor(w_query, dtype=torch.float32)
    w_value = torch.tensor(w_value, dtype=torch.float32)

    # get K, Q,V

    keys = torch.tensor(np.dot(x,w_key) ,dtype=torch.float32)
    querys = torch.tensor(np.dot(x,w_query),dtype=torch.float32)
    values = torch.tensor(np.dot(x,w_value),dtype=torch.float32)

    # get attention scorce
    attention_scores = torch.tensor(np.dot(querys,keys.T))
    print(attention_scores)
    # 计算soft-max
    attention_scores_softmax = torch.tensor( softmax(attention_scores,dim=-1) )
    print(values.shape)
    weight_values = values[:,None] * attention_scores_softmax.T[:,:,None]
    outputs = weight_values.sum(dim=0)
    return outputs

if __name__ == "__main__" :
    b = preData()
    print(b)

reference

• source of code