DeepSpeed combined with Megatron-LM training GPT2 model notes (on) - Code World

DeepSpeed combined with Megatron-LM training GPT2 model notes (on)

News 2023-09-07 01:34:30 views: null

NoSuchKey

Guess you like

Origin blog.csdn.net/just_sort/article/details/131173500

DeepSpeed combined with Megatron-LM training GPT2 model notes (on)

[DeepSpeed Tutorial Translation] II, Megatron-LM GPT2, Zero Redundancy Optimizer and ZeRO-Offload

PTM: Introduction to large model acceleration methods or frameworks (pre-training stage/inference stage), commonly used frameworks (Megatron-LM/Colossal-AI/DeepSpeed, etc., FastLLM/vLLM, etc.), detailed strategies for case applications

DeepSpeed accelerates large model training

GPT series training and deployment - GPT2 environment configuration and model training

A detailed understanding of the GPT2 model structure and its training process—GPT series training and deployment

Custom model and data for DeepSpeed-Chat training

[BBuf's cuda study notes ten] Megatron-LM's gradient_accumulation_fusion optimization

About the training verification generated by gpt2

U-Net combined with GAN model training

[Deep Learning] Framework for Large Model Training--Use of DeepSpeed

DeepSpeed: Large model training framework | JD Cloud technical team

Deep learning: Large-scale model distributed training framework DeepSpeed

Pre-training of large language models [2]: GPT, GPT2, GPT3, GPT3.5, GPT4 related theoretical knowledge and model implementation, model application and detailed explanation of the differences between versions

Notes - Model Training: Monitoring

[Paper notes] chatgpt series 2.3 DeepSpeed-chat SFT training

Notes - model training: Regular Loss

[NLP] In-depth understanding of Megatron-LM

Notes - model training: Save reading using the model

ColossalAI GPT2 distributed training debugging configuration - GPT series training and deployment

ZeRO & DeepSpeed: allows training model has more than 100 billion parameter optimization (Microsoft)

[Translation] DeepSpeed: A very large-scale model training tool that everyone can use

Popular understanding of Megatron-DeepSpeed: the technology behind the 100 billion parameter model BLOOM

[LLM] DeepSpeed distributed training framework

v-model｜ Youth training camp notes

Fudan University released the low-memory optimization technology LOMO | It reduces the memory usage of large model training to 10.8%, which is far ahead of DeepSpeed!

TensorFlow combined training data

"Machine Learning in Practice: Based on Scikit-Learn, Keras and TensorFlow Version 2" - Study Notes (4): Training Model

[Megatron-DeepSpeed] Tensor parallel tool code mpu detailed explanation (2): encapsulation mappings of Collective communication operation

Distributed parallel training (DP, DDP, DeepSpeed)

Recommended

Ranking

Java Design Pattern Template Pattern (Template) Part 2

html5 placeholder attribute compatible ie11

Der Unterschied zwischen Docker und Podman

Extended Thinking of Visual Recognition Algorithms - How to Better Deepen the Understanding of Visual Recognition Algorithms

After more than a month, I finally saw an offer, an Android face-to-face experience

JAVA面向对象之——包装类；入门讲解进来就能看懂

content+animation to achieve loading effect

1. HTML+DIV+CSS Zero Basic Quick Start to Making Enterprise Station Video Course _17 css box model [margin calculation]

js call video playback duration

Static modeling (class diagram and object diagram 2)

Daily

More

2025-02-06(0)

2025-02-05(0)

2025-02-04(0)

2025-02-03(0)

2025-02-02(0)

2025-02-01(0)

2025-01-31(0)

2025-01-30(0)

2025-01-29(0)

2025-01-28(0)