21 common Transformer interview questions (GPT4 helps you get it done)

Recently, I saw an interview question about Transfomer . It was written very well. The fly in the ointment is that the answer is a bit simple. It just so happens that my PLUS has not expired, so I want to ask the omnipotent GPT4 to answer the question. What's the problem? Welcome to leave a message and private message me so that I can modify it in time, so that I can't mislead my children.

Let's take a look!

Click here to skip the table of contents and jump to the text

1. Why does Transformer use a multi-head attention mechanism? (why not use a header)

Article Answer:

Multiple heads can make the parameter matrix form multiple subspaces. The overall size of the matrix remains unchanged, but the dimension size corresponding to each head is changed. This allows the matrix to learn various information, but the calculation amount is similar to that of a single head.

GPT4 Answer:

The main reasons why Transformer uses the multi-head attention mechanism are as follows:

1. Diversity: The multi-head attention mechanism allows the model to simultaneously focus on different positional and semantic information. Each head can learn a different attention distribution to capture different dependencies in the input sequence. If only one header is used, the model may miss some important dependencies.
2. Richer Representations: Through the multi-head attention mechanism, the model can learn multiple self-attention representations in parallel. This allows the model to better capture the local and global information of the input data. In natural language processing tasks, such rich representations often lead to better performance.
3. Parallel Computing: The multi-head attention mechanism can take full advantage of the parallel computing capabilities of modern hardware, such as GPUs. Multiple heads can be processed simultaneously in one calculation, which can significantly improve computational efficiency.
4. Interpretability: Multi-head attention mechanisms can provide more insight into the inner workings of models. For example, it is possible to understand how the model pays attention to different input positions by examining the attention weights of each head. This interpretability helps analyze and improve model performance.
   In summary, the multi-head attention mechanism allows the Transformer model to better capture the complex dependencies of the input data, improve the representation ability of the model, make full use of parallel computing capabilities, and provide better interpretability. Although single-head attention mechanisms can also achieve similar functions, multi-head attention mechanisms tend to achieve better results in practice.

2. Why does Transformer use different weight matrices to generate Q and K? Why can’t it use the same value for its own point multiplication?

Article Answer:

The initial weight of the request and the key value is different to solve the problem that the length of the input sentence may not be consistent with the length of the output sentence. And if the QK dimensions are the same, if you directly multiply K and K points without Q, you will find that the attention score matrix is ​​a symmetric matrix. Because it is the same matrix, they are all projected into the same space, so the generalization ability is very poor.

GPT4 Answer:

In the self-attention mechanism of the Transformer model, Q (query matrix), K (key matrix) and V (value matrix) are generated using different weight matrices for the following reasons:

1. Parameter separation: Using different weight matrices allows the model to learn different representations for queries, keys, and values ​​separately. This helps the model distinguish different information in the input sequence, thereby capturing more complex dependencies. If the same weight matrix is ​​used, it may limit the representation space learned by the model, resulting in performance degradation.
2. Different purposes: queries, keys, and values ​​play different roles in the self-attention mechanism. The query matrix is ​​used to retrieve relevant information, the key matrix is ​​used to represent information in the input data, and the value matrix is ​​used to summarize this information. Using different weight matrices allows the model to learn specific representations for each type of matrix, thereby optimizing the performance of the self-attention mechanism.
3. More powerful representation capabilities: Using different weight matrices allows the model to learn richer representations. In this way, the model can focus on the input data from multiple perspectives, which can better capture the dependencies in the input sequence. If only one weight matrix is ​​used, the model may not be able to learn enough information, resulting in poor performance.
   Although in some cases, similar functions may be achieved by using the same weight matrix, it has been proved that using different weight matrices can help the Transformer model better capture the complex dependencies of the input data, thereby improving the representation ability of the model and performance.

3. Why does Transformer choose dot multiplication instead of addition when calculating attention? What is the difference in computational complexity and effect between the two?

Article Answer:

The dot product of K and Q is to obtain an attention score matrix, which is used to purify V. K and Q are calculated using different W_k and W_Q, which can be understood as projections on different spaces. Because of the projection of different spaces, the expressive ability is increased, and the generalization ability of the calculated attention score matrix is ​​higher.

GPT4 Answer:

In Transformer, the self-attention mechanism uses dot multiplication instead of addition for the following reasons:

1. Computational efficiency: Dot product can effectively measure the similarity between two vectors. When calculating attention weights, dot multiplication can calculate the similarity of the entire sequence at one time through matrix multiplication, thereby improving computational efficiency. The addition operation needs to be calculated separately for each component of the vector, which may lead to lower computational efficiency in some cases.
2. Expressiveness: The dot product operation can capture the similarity between the query vector and the key vector, allowing the model to focus on the most relevant part of the query vector. In contrast, the addition operation may not capture the similarity between two vectors well, causing the model to focus on less relevant information.
   In terms of computational complexity, the complexity of dot product attention and additive attention is not much different. The complexity of dot product attention is O(n^2 * d), where n is the sequence length and d is the vector dimension; while the complexity of additive attention is O(n * d). In practical applications, the sequence length n is usually much larger than the vector dimension d, so the difference in computational complexity between the two is not significant.
   In effect, it has been proved that dot product attention outperforms additive attention in many tasks. The dot product can better capture the similarity between vectors, allowing the model to focus on the most relevant part of the query vector. However, additive attention also has its applicability, such as in tasks that deal with sparse data or require attention to specific attributes.
   In conclusion, although dot-product attention and additive attention have little difference in computational complexity, in the Transformer model, dot-product attention performs better in capturing inter-vector similarity and actual performance.

4. Why do you need to scale attention before performing softmax (why divide by the square root of dk), and explain it using formula derivation

Article Answer:

Suppose Q and K have mean 0 and variance 1. Their matrix product will have mean 0 and variance dk, so using the square root of dk is used for scaling, because the matrix product of Q and K should have mean 0 and variance 1, so that a smoother The softmax. When the dimension is large, the dot product result will be large, which will cause the gradient of softmax to be small. To mitigate this effect, the dot product is scaled.

GPT4 Answer:

In Transformer's self-attention mechanism, the attention matrix obtained by point multiplication needs to be scaled before softmax (divided by $d_k$The reason for the square root of ) is to prevent the result of the dot product from being too large, causing the gradient of the softmax function to disappear. Below we explain this process in detail.
set $Q$ and$K$ is the query matrix and the key matrix, and their dimensions are$(n,d_k)$ . Attention Matrix$The\; calculation\; process\; of\; A$ is as follows:

1. $A=Q{K}^T$ , of which$The\; dimension\; of\; A$ is$(n,n)$。
2. Against $Each\; element\; of\; A$ is scaled, that is,$A=\frac{A}{\sqrt{d_k}}$。
3. Against AAEach row of$A\; applies\; the\; softmax\; function.$
   In calculating$When\; A$ ,$Q$ and$A\; dot\; product\; of\; K$ may result in larger values. when$d_k$When increasing, the variance of the dot product result will also increase. This is because each component of the vector is sampled from some distribution, and the result of their dot product becomes larger as the dimensionality increases. This results in AAThe values ​​in$A\; can\; become\; very\; large.$
   WhenAAWhen the values ​​in$A\; are\; large,\; applying\; the\; softmax\; function\; may\; lead\; to\; the\; vanishing\; gradient\; problem.$The softmax function is defined as follows:
   $$\text{softmax}(x{)}_i=\frac{\exp (x_i)}{{\sum }_j\exp (x_j)}$$
   To $x_i$When the value of is large, $\exp \left(x_i\right)$ can become very large, causing the output of the softmax function to approach 0 or 1. In this case, the gradient of the softmax function becomes very small, leading to a vanishing gradient problem during training.
   To solve this problem, we need to$A$ is scaled, that is, divided by$\sqrt{d_k}$. This makes $The\; values\; in\; A$ become smaller, thereby alleviating the vanishing gradient problem. In practice, this scaling strategy is proven to be effective, improving the performance of Transformer models.

5. How to do mask operation on padding when calculating attention score?

Article Answer:

Set the position where the mask needs to be negative infinity, and then add the attention score

GPT4 Answer:

When calculating the attention score, the purpose of masking the padding is to ensure that the model does not pay attention to invalid (padding) positions in the input sequence. This is usually achieved by setting the value of the corresponding filled position in the attention score matrix to negative infinity before computing the softmax.
The following are the specific steps to perform mask operation on padding:

1. Creates a binary mask matrix from the length of the input sequence. Each element in the sequence is either 1 (non-fill position) or 0 (fill position). Assuming that the length of the input sequence is n, the dimension of the mask matrix is ​​(n, n).
2. Replace 0s in the binary mask matrix with negative infinity (eg: float('-inf')). In this way, the attention scores of the filled positions will become very small values ​​before the softmax calculation.
3. Add the mask matrix to the original attention score matrix (scaled A matrix). This will make the score of the filled position approach negative infinity.
4. Apply a softmax function to the updated attention score matrix. Since the score of the filling position is close to negative infinity, after softmax calculation, their corresponding weights will be close to 0, so as to ensure that the model will not pay attention to the filling position.
   This mask operation can prevent the model from paying attention to irrelevant filling positions in the self-attention calculation, thereby improving the performance of the model when processing variable-length sequences.

6. Why is it necessary to reduce the dimensionality of each head when performing multi-head attention?

Article Answer:

Transform the original high-dimensional space into multiple low-dimensional spaces and finally stitch them together to form an output of the same dimension, thereby enriching the characteristic information and reducing the amount of calculation.

GPT4 Answer:

When performing multi-head attention, the output of each head requires dimensionality reduction for several reasons:

1. Number of parameters and computational complexity: The output dimension of each head is (n, d_v), where n is the sequence length and d_v is the value vector dimension of each head. After concatenating the output of all heads, the dimension becomes (n, h * d_v), where h is the number of heads. If the dimensionality is not reduced, the number of parameters and computational complexity of the model will increase significantly, which may lead to problems such as overfitting and reduced computational efficiency.
2. Keep input and output consistent: In Transformer, both the input and output of the multi-head attention layer need to match other layers (such as feed-forward neural network layers). In order to maintain the consistency of input and output dimensions, the output of multi-head attention needs to be dimensionally reduced to make it the same as the input and output dimensions of other layers.
3. Fusion of multi-head information: The dimensionality reduction operation is actually to fuse the information of different heads together to extract the common features of each head. This helps to improve the expressive power of the model, enabling it to capture richer semantic information.
   Dimensionality reduction operations are usually implemented through linear transformations. After the output of multi-head attention is spliced, we use a weight matrix W_o (dimension (h * d_v, d_model)) to linearly transform the spliced ​​output to obtain a dimensionality-reduced result. Here d_model is the model dimension of Transformer. In this way, the multi-head attention layer is able to capture rich information in the input sequence while keeping the computational complexity low.

7. Tell me about Transformer's Encoder module?

Article Answer:

Input embedding - plus positional encoding - multiple encoder layers (each encoder layer contains a fully connected layer, a multi-head attention layer, and a point feedforward network layer (including an activation function layer))

GPT4 Answer:

Transformer's Encoder module is composed of a series of identical layers, each layer contains two main sublayers: Multi-Head Self-Attention layer and Feed-Forward Neural Network layer. In addition, there is a residual connection (Residual Connection) and layer normalization (Layer Normalization) operation after each sublayer.
The following are the main components of Transformer's Encoder module:

1. Word Embedding: First, each word of the input sequence is mapped to a fixed-dimensional vector as the input of the model.
2. Positional Encoding: In order for the model to capture sequential information in the sequence, the positional encoding needs to be added to the word embedding. Positional encoding is a method of representing the position of a word in a sequence and can be implemented using different techniques such as sine and cosine functions.
3. Multi-Head Self-Attention: This layer enables the model to focus on different positions in the input sequence, thereby capturing contextual information. In multi-head self-attention, the input is divided into multiple heads, and the attention weights of each head are calculated separately, and then the output of each head is concatenated and dimensionally reduced.
4. Residual Connection: Add the input vector itself to the output of multi-head self-attention, that is, implement a skip connection. This helps avoid the vanishing gradient problem in deep networks and improves training stability.
5. Layer Normalization: Normalizes the residual connected output to have zero mean and unit variance. This helps to improve the stability and convergence speed of the training process.
6. Feed-Forward Neural Network: This is a simple fully connected neural network responsible for learning non-linear representations from features extracted from the attention layer.
7. Residual connections and layer normalization are performed again.
   The entire Encoder module is stacked by multiple such layers. In practical applications, the number of layers can be adjusted according to the complexity and requirements of the task. Finally, the output of the Encoder is used as the input of the Transformer's Decoder module to generate the target sequence.

8. Why do I need to multiply the matrix by the root of the embedding size after obtaining the input word vector?

Article Answer:

The initialization method of the embedding matrix is ​​xavier init. The variance of this method is 1/embedding size, so multiplying the root of the embedding size makes the variance of the embedding matrix 1. This scale may be more conducive to the convergence of the embedding matrix.

GPT4 Answer:

I now understand that what you are asking is why you need to multiply by the root of the word embedding dimension (embedding size) when initializing the word embedding matrix. The reason here really has to do with the way you mentioned initialization.
In a neural network, the initialization method of the weight matrix has an important impact on the stability and convergence speed of the training process. Xavier initialization (also known as Glorot initialization) is a commonly used initialization method that adjusts the initial value of the weight matrix according to the number of input and output neurons. The purpose of this method is to make the output of each layer have similar variance, so as to avoid the problem of gradient explosion or gradient disappearance.
The specific operation of Xavier initialization is to start from a mean value of 0 and a variance of $\frac{1}{n}$Sampling weights in a normal distribution of , where $n$ is the number of input neurons. In the case of word embedding matrices, the number of input neurons is equal to the word embedding dimension. Therefore, we need to multiply the weight matrix by the root of the word embedding dimension so that its variance is 1. In this way, at this scale, the initialization of the word embedding matrix is ​​more conducive to the convergence and training stability of the model.
In short, the purpose of multiplying the word embedding matrix by the root of the word embedding dimension is to adjust the initial value of the matrix according to the Xavier initialization method to improve the training stability and convergence speed of the model.

9. Briefly introduce Transformer's position encoding? What's the point and pros and cons?

Article Answer:

Because self-attention is position-independent, no matter what the order of the sentence is, the hidden embedding of the token calculated by self-attention is the same, which obviously does not conform to human thinking. Therefore, there must be a way to express the position information of a token in the model. The transformer uses a fixed positional encoding to represent the absolute position information of the token in the sentence.

GPT4 Answer:

Transformer's Positional Encoding is a way to inject information about the position of words in a sequence. Because Transformer's self-attention mechanism is disordered, it cannot directly capture the order information of words in the sequence. Positional encoding solves this problem, enabling the model to distinguish words in different positions.
A common method of position encoding is to use fixed-dimensional vectors generated by the sine and cosine functions. Given a position $p$ and a dimension$i$ , the positional encoding is calculated using the following formula:
$$PE(p,2i)\_=\mathrm{without}(p/10000^{2i/d_{model}})$$
$$PE(p,2i\_+1)=\cos (p/10000^{2i/d_{model}})$$
where$p$ is the position of the word in the sequence,$i$ is the dimension index in the position encoding vector,$d_{model}$is the model dimension of Transformer. This approach can produce unique vector representations for different locations while maintaining learnable and differentiable properties.
The significance of positional encoding is to provide a way to capture the order information of the input sequence. This is crucial for many natural language processing tasks (such as translation, question answering, etc.), because the output of these tasks often depends on the order of words in the input sequence.
advantage:

1. No learning required: Relative to learnable positional representations such as positional embeddings, positional encodings require no additional training, reducing model complexity.
2. Uniqueness: Positional encoding based on sine and cosine functions can generate unique vector representations that help the model distinguish words in different positions.
3. Scalability: This approach can generate positional encodings of arbitrary length, enabling the model to handle input sequences of varying lengths.

shortcoming:

1. Fixed representations: Positional encodings are fixed and cannot be adjusted to the training data like learnable positional embeddings can. This can cause performance limitations in some tasks.
2. Failure to capture fine-grained order relationships: Positional encodings generated by sine and cosine functions may fail to capture complex order patterns in the input sequence, especially in higher dimensions.
   Although positional encoding has certain limitations, it has been shown to be an effective way to capture order information in sequences in Transformer models.

10. What other techniques do you know about position encoding, and what are their advantages and disadvantages?

Article Answer:

Relative Position Encoding (RPE) 1. When calculating the attention score and weighted value, add a trainable parameter representing the relative position. 2. When generating multi-head attention, the absolute position of the key is converted into a position relative to the query. 3. The complex domain function, knowing the word vector representation of a word at a certain position, can calculate its position at any position Word vector representation. The first two methods are word vector + position encoding, which belong to the remedy for a dead sheep. The plural domain is to generate the corresponding position information when the word vector is generated.

GPT4 Answer:

In addition to positional encoding based on sine and cosine functions, there are other techniques on positional encoding, as follows:
1). Learned Positional Embedding:
This method treats positional encoding as a learnable The parameter matrix of . During training, the model learns an appropriate position representation according to the demands of the task. This approach is commonly used in Recurrent Neural Networks (RNN) and Transformer-based models such as BERT.
advantage:

• Adaptability: Positional embeddings can be adaptively tuned to the training data to capture more complex sequential patterns.
• Flexibility: Can be co-trained with other parts of the model to achieve better performance.
  shortcoming:
• Additional training required: Positional embeddings increase the number of parameters of the model, potentially leading to overfitting and longer training times.
• Length limitation: The pre-defined positional embedding matrix limits the maximum length of the input sequence.
  2). Diagonal Positional Encoding:
  This method uses a diagonal matrix to represent positional information. Diagonal position encoding uses matrix multiplication to multiply each word in the input sequence with the corresponding position representation.
  advantage:
• Simple: Relatively simple to implement and compute.
• No learning required: Similar to positional encoding based on sine and cosine functions, no additional training is required.
  shortcoming:
• Limited expressiveness: Diagonal position encodings may fail to capture complex sequential patterns.
• Sparsity: Diagonal position encodings may lead to sparse representations, affecting model performance.
  3). Axial Positional Encoding:
  This method decomposes positional information into multiple axes (for example, rows and columns), and then adds the positional encodings of different axes. This approach is more common in image processing tasks, such as Transformer-based image classifiers (such as ViT).
  advantage:
• Efficient: Decomposing positional encodings can reduce computational complexity.
• Scalability: Can handle multi-dimensional data such as images and videos.
  shortcoming:
• Applicability: Axis-angular position encoding is mainly used for processing multi-dimensional data and may not be suitable for all tasks.
  Various position encoding techniques have different advantages and disadvantages. Choosing an appropriate position encoding method depends on the requirements of the task and the characteristics of the model.

11. Briefly talk about the residual structure and meaning in Transformer.

Article Answer:

Both the self-attention layer and the ffn layer of the encoder and decoder have residual connections. The gradient will not disappear during backpropagation.

GPT4 Answer:

In the Transformer model, the residual structure (Residual Connection) is an important component, which helps to improve the training stability and performance of the model. The residual structure is a skip connection, which can directly add the input to the output of the layer, thereby preserving the original information in the deep network.
In Transformer, the residual structure usually appears in the following two parts:

1. After self-attention (Self-Attention): After calculating the multi-head self-attention, the output of attention will be added to the input (original word vector plus position encoding) to form a residual connection. Then, the result is passed to the subsequent layer normalization (Layer Normalization) and feed-forward neural network.
2. After Feed-Forward Neural Network: After the feed-forward neural network is calculated, its output is added to the layer normalization result after self-attention to form another residual connection. After layer normalization again, pass the result to the next encoder layer or decoder layer.
   The meaning of the residual structure:
3. Alleviating the Vanishing Gradient Problem: The vanishing gradient problem is a common challenge in deep neural networks. The residual structure allows gradients to more easily propagate through multiple layers during training, helping to alleviate the vanishing gradient problem.
4. Faster convergence: Since the residual structure preserves the original input information, the model can learn the identity mapping more easily. This can speed up the convergence of the model and help the model reach a lower error faster during training.
5. Enhance the expressive ability of the model: the residual structure can enable the model to capture more levels of information during the training process. This combination helps to improve the expressiveness of the model, leading to better performance.

12. Why does the transformer block use LayerNorm instead of BatchNorm? Where is LayerNorm in Transformer?

Article Answer:

Between the multi-head attention layer and the activation function layer. CV uses BN because it believes that the information of the channel dimension is of great significance to the cv aspect. If the channel dimension is also normalized, it will cause a certain loss of information of different channels. In the same way, the nlp field believes that the sentence length is inconsistent, and the information of each batch has nothing to do with it, so only the normalization of the information in the sentence is considered, that is, LN.

GPT4 Answer:

In Transformer, use Layer Normalization (layer normalization) instead of Batch Normalization (batch normalization) for the following reasons:

1. Variable length sequences: Input sequences in natural language processing tasks are often of variable length. Unlike batch normalization, layer normalization is performed on a single sample and thus handles sequences of variable length more naturally.
2. No batching required: Batch normalization requires calculating the mean and variance over the entire batch, which can lead to different statistics being used during the training and inference phases. Layer normalization computes the mean and variance over different dimensions of a single sample, thus using the same statistics during training and inference.
3. Self-attention mechanism: The self-attention mechanism in Transformer allows words in each position to interact with words in other positions, so it is not suitable for batch normalization. Layer normalization can better accommodate this interaction pattern.
   The position of Layer Normalization in Transformer: In Transformer's encoder and decoder modules, layer normalization mainly occurs in the following two places:
4. After self-attention (Self-Attention): After calculating the multi-head self-attention, the output of attention will be added to the input (original word vector plus position encoding) to form a residual connection. Then, pass the result to layer normalization. This layer normalization normalizes the sum of input and attention output, and then passes the normalized result to the subsequent feedforward neural network.
5. After the feed-forward neural network (Feed-Forward Neural Network): After the feed-forward neural network is calculated, its output is added to the layer normalization result after self-attention to form another residual connection. This residual connection is then layer-normalized before passing the result to the next encoder or decoder layer.
   By using layer normalization in key parts, the Transformer model can maintain stable gradient propagation and better performance during training.

13. Briefly talk about BatchNorm technology and its advantages and disadvantages.

Article Answer:

Batch normalization is to normalize each batch of data before entering the activation function, which can improve the convergence speed, prevent over-fitting, prevent the gradient from disappearing, and increase the sensitivity of the network to data.

GPT4 Answer:

Batch Normalization is a technique used to speed up neural network training and improve model performance. It was proposed by Ioffe and Szegedy in 2015, and its main purpose is to solve the problem of internal covariate shift in the training process of deep neural network.
The core idea of ​​batch normalization is to normalize the input of the neural network layer in each training batch. Specifically, it calculates the mean and variance of the current batch of data along each feature dimension, and then uses these statistics to standardize the data. Afterwards, batch normalization introduces two learnable parameters (scaling factor and offset factor), which are used to adjust the normalized data to have appropriate scaling and offset.
advantage:

1. Speeds up convergence: Batch normalization helps speed up the convergence of neural networks because it reduces the risk of vanishing and exploding gradients, allowing larger learning rates to be used.
2. Reduce sensitivity to initialization: Batch normalization can reduce the sensitivity of the model to the way the weights are initialized by normalizing the input of each layer.
3. Regularization effect: Batch normalization has a regularizing effect to some extent, because it introduces some noise (due to statistical differences between batches), which helps prevent overfitting.

shortcoming:

1. Batch dependency: Batch normalization depends on the batch size and the distribution of data across batches. During the training and inference stages, the data distribution may be different, which may lead to performance degradation.
2. Variable sequence length problem: Batch normalization is not suitable for variable length sequence data because it needs to calculate statistics on the entire batch. This can cause problems in natural language processing tasks.
3. Computational overhead: While batch normalization can speed up convergence, it increases the computational overhead of each layer, which slows down the inference speed of the entire network.
   In summary, batch normalization is a technique that has significant implications in neural network training. It helps speed up convergence, reduces sensitivity to initialization, and has some regularization effects. However, it also suffers from some disadvantages, such as batch dependency, variable sequence length issues, and computational overhead.

14. Briefly describe the feedforward neural network in Transformer? What activation function was used? Related pros and cons?

Article Answer:

Input embedding - plus positional encoding - multiple encoder layers (each encoder layer contains a fully connected layer, a multi-head attention layer, and a point feedforward network layer (including an activation function layer)) - multiple decoder layers (each Encoder layer consists of fully connected layer, multi-head attention layer and point feedforward network layer) - fully connected layer, using the relu activation function

GPT4 Answer:

In the Transformer model, the Feed-Forward Neural Network (FFNN) is an important component. It is located after the Self-Attention module as a sublayer of the encoder and decoder. The feed-forward neural network processes the vectors for each position in the sequence independently, providing the model with additional non-linear representation capabilities.
The feedforward neural network in Transformer is usually a two-layer fully connected network with the following structure:

1. The first layer of fully connected network (linear layer): maps the input vector to a higher dimensional hidden space. This process can be seen as an extension of the input features. Activation function: Use a nonlinear activation function between two linear layers, usually ReLU (Rectified Linear Unit) or GELU (Gaussian Error Linear Unit).
2. The second layer of fully connected network (linear layer): maps the output of the activation function back to the dimensions of the original input vector. This layer essentially compresses the high-dimensional representation in the hidden space to generate new feature representations.
   In the original Transformer paper, the activation function used was ReLU. ReLU is a simple and effective non-linear activation function defined as: f(x) = max(0, x). The advantages of ReLU include computational simplicity, fast convergence, and the ability to alleviate the vanishing gradient problem. However, a major disadvantage of ReLU is that it has zero gradient in negative intervals, which can lead to the "dead ReLU" problem, where some neurons no longer update.
   GELU is an activation function that has become popular in natural language processing tasks in recent years. It can be seen as a smoothed version of ReLU with better gradient distribution. GELU is calculated as: f(x) = 0.5x(1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3))). Compared with ReLU, GELU has non-zero gradient in the negative interval, so it can alleviate the "dead ReLU" problem. However, the computational complexity of GELU is high, and it will increase some computational overhead compared to ReLU.
   In summary, the feedforward neural network in Transformer is a two-layer fully connected network, usually using ReLU or GELU as the activation function. The feed-forward neural network provides the model with additional non-linear representation ability, which helps to improve the performance of the model.

15. How do the Encoder and Decoder interact?

Article Answer:

By transposing the seq_len dimension and depth dimension of encoder_ouput, perform matrix multiplication twice, that is, q kT v output can get the output of target_len dimension

GPT4 Answer:

In the Transformer model, the key part of the mutual interaction between the Encoder and the Decoder is realized through the Encoder-Decoder Attention Layer in the Decoder. Specifically, the interaction process is as follows:

1. First, the Encoder side receives source sequences (e.g., source language sentences) as input and processes them through a series of encoder layers. Each encoder layer contains multi-head self-attention and feed-forward neural networks. After processing all encoder layers, the Encoder side outputs a context vector representing the source sequence.
2. Next, the Decoder side receives the target sequence (for example, part of the target language sentence) as input. The target sequence is first processed through a series of decoder layers. Similar to the encoder layer, each decoder layer also contains multi-head self-attention, encoder-decoder attention layer and feed-forward neural network.
3. In the encoder-decoder attention layer of the decoder, the Decoder side utilizes the context vector from the Encoder side to interact with the current decoder state. Specifically, the encoder's output (i.e., the context vector) serves as "key" and "value", while the decoder's self-attention output serves as "query". In this way, the decoder can adapt the representation of the target sequence based on the information of the source sequence.
4. Finally, the output of the decoder is passed to a linear layer and a softmax layer to generate the next word in the target sequence. During the training phase, the model calculates the loss and optimizes it based on the actual words of the target sequence. During the inference phase, the model generates words one at a time until a termination condition is reached (such as generating a specific terminator or reaching a maximum sequence length).
   In summary, the Encoder and Decoder sides interact through the encoder-decoder attention layer in the Transformer model. This interaction mechanism allows the model to generate more accurate target sequences based on information from source sequences.

16. What is the difference between the multi-head self-attention in the Decoder stage and the multi-head self-attention in the encoder?

Article Answer:

Decoder has two layers of mha, and encoder has one layer of mha. The second layer of mha of decoder is to convert input and output sentence lengths. Decoder's request q can be different from the penultimate dimension of key k and value v, but encoder's qkv Dimensions are the same.

GPT4 Answer:

In the Transformer model, the multi-head self-attention mechanisms of the encoder (Encoder) and decoder (Decoder) are the same in terms of calculation methods, but their application scenarios and constraints are different. The main differences are:

1. Masking: In the multi-head self-attention layer of the Decoder, in order to prevent the output of the current position from being affected by future position information, we need to apply a mask to the attention matrix. This is done by creating a lower triangular matrix (upper triangular part is negative infinity) and adding it to the attention score matrix, followed by a softmax calculation. This way, the decoder can only consider previous words (including the word at the current position) when generating each word, but cannot see future words. In the multi-head self-attention layer of the encoder, there is no need for this masking operation, since all information in the source sequence is known.
2. Input sequence: The multi-head self-attention of the encoder (Encoder) receives the source sequence as input, and processes the relationship between words in the input sequence and other words. The multi-head self-attention of the decoder (Decoder) receives the target sequence as input, and processes the relationship between the words in the target sequence and other words when generating each word of the target sequence. Although the calculation methods are the same, the sequences that these two self-attention mechanisms focus on are different.
3. Purpose: The multi-head self-attention of the encoder (Encoder) is mainly used to capture the global dependencies in the source sequence in order to better represent the input. The multi-head self-attention of the decoder (Decoder) is used to capture the global dependencies in the target sequence, so that the context information of the target sequence can be considered when generating new words.
   In summary, multi-head self-attention for encoder and decoder is computationally the same, but their application scenarios, constraints and input sequences are different. The decoder's multi-head self-attention requires masking to prevent the output of the current position from being affected by future position information, while the encoder's multi-head self-attention does not need to perform such masking.

17.Where is the parallelization of Transformer?

Article Answer:

The parallelization of Transformer is mainly reflected in the self-attention module. On the Encoder side, the Transformer can process the entire sequence in parallel, and obtain the output of the entire input sequence through the Encoder side, but rnn can only be executed from front to back.

GPT4 Answer:

The parallelization of Transformer is mainly reflected in the following aspects:

1. Multi-Head Attention: In the multi-head self-attention mechanism, the attention distribution of each head is calculated independently, and these heads can be calculated simultaneously. This design makes full use of the parallel computing capability of modern computing hardware (such as GPU), thereby improving the computational efficiency of the model.
2. Positional Encoding: Since the Transformer model does not have a built-in loop structure (such as RNN), it can simultaneously process words in all positions given an input sequence. In the input stage, the model infuses positional information for each positional word by adding positional encodings to the word vectors. This allows the model to process words in all positions in parallel as it processes the sequence.
3. Stacking of encoder and decoder layers: Both the encoder and decoder in the Transformer model are stacked by multiple layers of networks with the same structure. During training and inference, each layer can be independently computed, enabling layer-to-layer parallelization. However, in practice, in order to achieve efficient gradient propagation and weight updating, skip connections (such as residual connections and layer normalization) are usually added between layers to form an overall computational graph.
4. Parallelization of the training phase: During the training phase, the model can use the mini-batch training method for parallel computing. By combining multiple training samples into a batch, the model can process these samples simultaneously, thus taking full advantage of the parallelism of computing hardware. It should be noted that when using this method, it may be necessary to pad the input sequence to keep the same length.
   These parallelization designs enable Transformer to efficiently compute on modern computing hardware while maintaining strong representation and generation capabilities.

18. Can the Decoder be parallelized?

Article Answer:

Yes during training, but not during interaction

GPT4 Answer:

On the Decoder side of the Transformer model, some parts can be parallelized, while other parts are limited to a certain extent.
In the training phase, since we usually use the "Teacher Forcing" strategy (that is, using the real target sequence as input instead of the output generated by the model), the multi-head self-attention layer on the Decoder side can process the entire target sequence in parallel. However, to prevent the output of the current position from being influenced by future position information, the multi-head self-attention layer of the decoder needs to apply a mask. This masking operation limits parallelism on the Decoder side, but still allows parallel computation across multiple samples in a batch.
In the inference stage (such as using greedy decoding or beam search), the parallelism on the Decoder side is more limited. This is because when generating each word of the target sequence, the model needs a previously generated word as input. In this case, the multi-head self-attention layer on the Decoder side cannot process the entire target sequence at the same time, so words must be generated sequentially. Nevertheless, it is still possible to process multiple different sequences simultaneously (such as searching for multiple candidate sequences in beam search) in one inference pass.
In short, in the training phase, the Decoder side can be parallelized to a certain extent, but it is limited by the mask operation. In the inference phase, the parallelism on the Decoder side is more limited because the words of the target sequence need to be generated sequentially. However, some degree of parallelism can still be achieved at the batch level and between different sequences.

19. Briefly describe the wordpiece model and byte pair encoding. Has it been used in practice?

Article Answer:

"Traditional word representation methods cannot handle unknown or rare vocabulary (OOV problem) well.
Traditional word tokenization methods are not conducive to the model learning the relationship between affixes."
BPE (byte pair encoding) or binary encoding is a simple data A compressed form in which the most common pair of consecutive bytes of data are replaced by bytes that do not exist in that data. A replacement table is required to reconstruct the original data for later use.
Pros: Can effectively balance vocabulary size and number of steps (number of tokens needed to encode a sentence).
Disadvantages: Based on greedy and deterministic symbol replacement, it cannot provide multiple fragmentation results with probabilities.

GPT4 Answer:

WordPiece Model (WPM) and Byte Pair Encoding (BPE) are both unsupervised segmentation methods for segmenting text into subword units. These two methods can alleviate the vocabulary size and out-of-vocabulary (OOV) problems to some extent.

1. WordPiece Model (WPM): WPM iteratively divides text into subword units. Initially, the vocabulary contains all words. Then at each iteration, WPM counts the frequencies of all possible subword pairs in the vocabulary and merges the most frequent subword pairs into a new subword. This process continues until the preset vocabulary size is reached. During tokenization, WPM splits the input text into the longest sequence of subwords that occurs in the vocabulary.
2. Byte Pair Encoding (BPE): BPE works very similar to WPM, but it was originally used for data compression. BPE also iteratively merges frequently occurring character pairs (or subword pairs) to create new subwords. The word segmentation process of BPE is the same as that of WPM, which divides the input text into the longest subword sequence that appears in the vocabulary. The main difference between BPE and WPM is their focus on vocabularies and merge operations. BPE focuses more on character-level merging, while WPM focuses more on subword-level merging.

Practical applications: BPE and WPM have been widely used in many natural language processing tasks, especially in pre-trained language models (such as BERT and GPT series models). These models use BPE or WPM to segment the text during the training phase, so as to learn better semantic representation at the subword level. In downstream tasks (such as text classification, named entity recognition, etc.), pre-trained models can better handle unknown words and reduce vocabulary size, thereby improving the generalization ability and efficiency of the model.

20. How is the learning rate set during Transformer training? How is Dropout set up and where is it located? Is there anything to pay attention to when testing Dropout?

Article Answer:

LN is to solve the problem of gradient disappearance, and dropout is to solve the problem of overfitting. Adding LN after embedding is beneficial to the convergence of embedding matrix.

GPT4 Answer:

In the Transformer training process, the learning rate and Dropout settings are very important, and they have a significant impact on the convergence speed and performance of the model.

1. Learning rate setting: Transformer training usually adopts an adaptive learning rate adjustment strategy, called "noam" learning rate scheduling. In this strategy, the initial learning rate is very low and increases linearly with the number of training steps until it reaches the preset maximum learning rate. Then, the learning rate gradually decreases as the number of training steps increases. This scheduling strategy can make the model adapt to the data faster in the early stage of training, while keeping the learning rate low in the later stage of training to ensure the stable convergence of the model.
   Specifically, the formula of the noam learning rate scheduling strategy is:
   $$lrate=d_{model}^{-0.5}\cdot \min (step\_nu{m}^{-0.5},step\_num\cdot warmup\_step{s}^{-1.5})$$
   where$d_{model}$is the dimension of the model, $step\_num$ is the current number of training steps,$warmup\_stepsisthe\; numberof$ warm-up steps$.$During the warm-up phase, the learning rate is proportional to the number of training steps; after the warm-up phase, the learning rate is inversely proportional to the square root of the number of training steps.
2. Dropout setting: Transformer uses the Dropout technique to reduce the risk of model overfitting. In the original Transformer paper, it is recommended to set the dropout rate to 0.1. Dropout layers are typically located in:
   • The output of the multi-head self-attention layer;
   • The output of the feed-forward neural network layer;
   • The result of adding positional encodings and word vectors.
3. During the testing phase, it is important to note that dropout is generally not used. In the model evaluation and inference phase, we want the model to provide stable and consistent output, so we need to turn off the dropout layer. In most deep learning frameworks, this can be achieved by setting the model into evaluation mode, such as used in PyTorch model.eval(). In evaluation mode, the dropout layer does not affect the output, thus ensuring the stability of the model output.

21. Why doesn't Bert's mask learn the technique of transformer to shield the score at attention?

Article Answer:

The goals of BERT and transformer are inconsistent. Bert is a language pre-training model that needs to fully consider the relationship between context, while transformer mainly considers the relationship between the i-th element in the sentence and the first i-1 elements.

GPT4 Answer:

BERT and Transformer adopt different strategies when dealing with masks. In BERT, the mask is mainly applied to the input sequence (ie Masked Language Model, MLM), while in the standard Transformer decoder, the mask is mainly applied to the self-attention mechanism of the decoder to prevent the leakage of future position information.
The main goal of BERT is to learn context-sensitive word representations by pre-training a deep bidirectional language model. To achieve this, BERT employs the Masked Language Model task. In this task, some input words are randomly replaced with special [MASK] tokens, and then the model is asked to predict these masked words. This masking strategy works directly on the input sequence instead of processing it in the attention mechanism. Therefore, the model structure of BERT is relatively simple, and there is no need to add a mask to the self-attention layer like the Transformer decoder.
On the other hand, the goal of the Transformer decoder is to generate a target sequence, which needs to protect the output of the current position from being affected by future position information. To achieve this, the decoder uses an upper-triangular mask in the self-attention mechanism, setting the attention score for future positions to negative infinity. In this way, when performing softmax calculation, the probability of future positions is close to zero, thereby realizing future information masking.
In summary, BERT and Transformer adopt different strategies when dealing with masks due to their different task goals and model structures. BERT directly applies a mask to the input sequence through the Masked Language Model task, while the Transformer decoder needs to add a mask in the self-attention mechanism to prevent the leakage of future position information.