1. Chatglm
Relatively simple, and the performance after fine-tuning is strange, you can refer to the deployment and fine-tuning tutorial of ChatGLM-6B
1.1 MNN deployment
https://github.com/wangzhaode/ChatGLM-MNN
1.1.1 Linux deployment
git clone https://github.com/wangzhaode/ChatGLM-MNN.git
(1) Compile MNN
cd MNN
mkdir build && cd build
#使用cuda
cmake -DCMAKE_BUILD_TYPE=Release -DMNN_CUDA=ON ..
make -j$(nproc)
cd ../..#退出
(2) File copy
cp -r MNN/include/MNN include
cp MNN/build/libMNN.so libs/
cp MNN/build/express/*.so libs/
(3) Weight download
and hang vpn
cd resource/models
# 下载fp16权值模型, 几乎没有精度损失
./download_models.sh fp16
# 下载int8权值模型,极少精度损失,推荐使用
./download_models.sh int8
# 下载int4权值模型,有一定精度损失
./download_models.sh int4
(4) experience
mkdir build && cd build
cmake -D WITH_CUDA=on ..
# start build(support Linux/Mac)
make -j$(nproc)
./cli_demo # cli demo
./web_demo # web ui demo
It probably looks like this, but it will report the memory soon, and it is also a problem they are currently solving
1.2 InferLLM deployment
https://github.com/MegEngine/InferLLM