Self-Distillation for Further Pre-training of Transformers - Code World

Self-Distillation for Further Pre-training of Transformers

Enterprise 2023-10-03 05:58:54 views: null

NoSuchKey

Guess you like

Origin blog.csdn.net/qgh1223/article/details/131624342

Self-Distillation for Further Pre-training of Transformers

【Paper Notes】BEIT:BERT PRE-TRAINING OF IMAGE TRANSFORMERS

Transformers pre-training model uses: Text Summary Summarization

Transformers pre-training model uses: translation Translation

[Notes] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

【论文笔记】BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

[Natural Language Processing | BERT] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Paper Explanation

Intensive reading of Li Mu's paper: BERT "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"

[Paper Notes] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Transformers pre-training model uses: Named Entity Recognition Named Entity Recognition

[NLP classic paper intensive reading] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Self-distillation of knowledge distillation [with code]

Effectiveness of self-supervised pre-training for speech recognition

Reading Literature 1: Bootstrapping ViTs: Towards Liberating Vision Transformers from Pre-training (article translation and its own understanding and summary)

DeiT：Training data-efficient image transformers & distillation through attention

Transformers pre-load training model | seven

[Paper reading] PointContrast self-supervised pre-training model semantic segmentation 2020

Introduction to multimodal pre-training + self-supervised learning + downstream tasks

Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentatio

SplitMask: Are large-scale datasets necessary for self-supervised pre-training?

A review of Nanyang Technological University's latest visual language model: pre-training, transfer learning and knowledge distillation have everything

ICML2021《Training data-efficient image transformers & distillation through attention》

The self-generated instruction framework Self-Instruct realizes the pre-training language model and instruction alignment (including practical operation + code)

CVPR 2022 | Tsinghua proposes Point-BERT: pre-training of point cloud self-attention model based on mask modeling

【论文笔记】Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds with Masked Occupan

The improvement for Bert is mainly reflected in increasing training corpus, adding pre-training tasks, improving mask methods, adjusting model structure, adjusting hyperparameters, model distillation, etc.

Retrieval scene pre-training

Rasa Course, Rasa Training, Rasa Interview, Transformers & Attention Self Attention of Rasa Practical Series

Interpretation of the paper: Self-Distillation from the Last Mini-Batch for Consistency Regularization

Domain Adaptation in CVPR 2023: Memory-efficient CoTTA via self-distillation regularization

Recommended

Ranking

Java Design Pattern Template Pattern (Template) Part 2

html5 placeholder attribute compatible ie11

Der Unterschied zwischen Docker und Podman

Extended Thinking of Visual Recognition Algorithms - How to Better Deepen the Understanding of Visual Recognition Algorithms

After more than a month, I finally saw an offer, an Android face-to-face experience

JAVA面向对象之——包装类；入门讲解进来就能看懂

content+animation to achieve loading effect

1. HTML+DIV+CSS Zero Basic Quick Start to Making Enterprise Station Video Course _17 css box model [margin calculation]

js call video playback duration

Static modeling (class diagram and object diagram 2)

Daily

More

2025-02-06(0)

2025-02-05(0)

2025-02-04(0)

2025-02-03(0)

2025-02-02(0)

2025-02-01(0)

2025-01-31(0)

2025-01-30(0)

2025-01-29(0)

2025-01-28(0)