[(Strongly recommended) Li Hongyi 2021/2022 Spring Machine Learning Course] 2022-How to Effectively Use Self-Supervised Models-Data-Efficient &Parameter-Efficient Tuning

Materials: pdf | Video

1. What is self-pretraining

Two training methods:

1. predict the next word
2. mask learning

fine-tune: embryonic stem cells, also need fine-tune to exert their power

2. There is a problem

Fine-tuning still requires some data The

model is too large

3. Data-Efficient Fine-tuning: Prompt Tuning (hint, more efficient use of data, such as when the amount of data is small)

The difference between Standard fine-tuning and Standard fine-tuning

Prompt Tuning will perform well in the case of small amount of data

less data

4. Data-Efficient Fine-tuning: Semi-supervised Learning (semi-supervised)

5. zero-shot

6. PLMs Are Gigantic -> Reducing the Number of Parameters (solution for large models)

Parameter-Efficient Fine-tuning

Adapter

LoRA (less parameters than Adapter (feature compression is smaller), faster inference (parallel insertion))

Prefix Tuning (core: fine-tuning is to change hidden representations)

Review Self-attention

Soft Prompting

4 Methods Summary

Early Exit (reduce inference time)

7. Closing Remarks