Article directory
- 1. What is self-pretraining
- 2. There is a problem
- 3. Data-Efficient Fine-tuning: Prompt Tuning (hint, more efficient use of data, such as when the amount of data is small)
- 4. Data-Efficient Fine-tuning: Semi-supervised Learning (semi-supervised)
- 5. zero-shot
- 6. PLMs Are Gigantic -> Reducing the Number of Parameters (solution for large models)
- 7. Closing Remarks
1. What is self-pretraining
Two training methods:
- predict the next word
- mask learning
fine-tune: embryonic stem cells, also need fine-tune to exert their power
2. There is a problem
Fine-tuning still requires some data The
model is too large
3. Data-Efficient Fine-tuning: Prompt Tuning (hint, more efficient use of data, such as when the amount of data is small)
The difference between Standard fine-tuning and Standard fine-tuning
Prompt Tuning will perform well in the case of small amount of data
less data
4. Data-Efficient Fine-tuning: Semi-supervised Learning (semi-supervised)
5. zero-shot
6. PLMs Are Gigantic -> Reducing the Number of Parameters (solution for large models)
Parameter-Efficient Fine-tuning
Adapter
LoRA (less parameters than Adapter (feature compression is smaller), faster inference (parallel insertion))
Prefix Tuning (core: fine-tuning is to change hidden representations)
Review Self-attention
Soft Prompting
4 Methods Summary
Early Exit (reduce inference time)
7. Closing Remarks