Download and install the ControlNet plug-in of Stable Diffuse AI painting and its corresponding model

Table of contents

2. Download and install the ControlNet plug-in

3. Download and install the ControlNet plug-in model

4. Other ways to download and install ControlNet plug-ins

5. ControlNet plug-in model name description

1. Brief introduction

Stable Diffusion is a text-to-image latent diffusion model created by researchers and engineers from CompVis, Stability AI, and LAION. It is trained using 512x512 images from a subset of the LAION-5B database. Using this model, any image including human faces can be generated. Because there is an open source pre-trained model, we can also run it on our own machine, as shown in the figure below.

Stable Diffusion is an AI drawing software (open source model), which can be deployed locally and switch between multiple models, and new models and open source libraries are updated and released every day. The most important thing is that it is free and has no limit on the number of drawings.

Github 网址：GitHub - AUTOMATIC1111/stable-diffusion-webui: Stable Diffusion web UI

To play Stable Diffusion, you must be able to play ControlNet. It is a very important extension plug-in (sd-webui-controlnet) of Stable Diffusion. It can not only be used to copy composition and human body posture, but also has many other powerful functions. To use ControlNet (sd-webui-controlnet), you must first learn how to install it.

ControlNet github address: https://github.com/lllyasviel/ControlNet

ControlNet-v1-1-nightly github address: https://github.com/lllyasviel/ControlNet-v1-1-nightly

sd-webui-controlnet
github address: https://github.com/Mikubill/sd-webui-controlnet

By the above way, ControlNet can reuse SD encoder as a deep, strong, robust and powerful backbone to learn different controls. A lot of evidence (like this and this) verifies that the SD encoder is an excellent backbone.

Note that the way we connect layers is computationally efficient. The original SD encoder does not need to store gradients (locked original SD encoder block 1234 and Middle). The required GPU memory is not much larger than the original SD, despite the many layers added.

All models and detectors can be downloaded from our Hugging Face page . Make sure the SD model is placed in "ControlNet/models" and the detector is placed in "ControlNet/annotator/ckpts". Make sure to download all necessary pretrained weights and detector models from our Hugging Face page , including the HED edge detection model, Midas depth estimation model, Openpose, and more.

2. Download and install the ControlNet plug-in

1. Run the Stable Diffuse WebUI, open the webpage, find the extension - available - click to load from, there will be many available extensions

NOTE: magic internet access may be required here

2. "Ctrl+F" Search for ControlNet, find the ControlNet plug-in, click to install

3. After installation, you can find the ControlNet plug-in in Extension-Installed

4. Check the ControlNet plug-in, then click Apply and restart the user interface. After the interface restarts, you can see the ControlNet related operation interface

5. In the extensions folder, you can see the ControlNet plug-in folder

3. Download and install the ControlNet plug-in model

1. Find the ControlNet plug-in download address on the hugging face

ControlNet plug-in model URL: lllyasviel/ControlNet-v1-1 at main

2. Description of the model name, some models are available and unavailable, and testing

When downloading, pay attention to the letter after " V11 " in the file name, p: means it can be used , e: means it is still in the test , u: means it is not completed , if the letter in the file name is e, do not download

3. The model is still relatively large, and you can download it according to your own needs. Here we take canny as an example

Note: Both the .pth and .yaml files of the corresponding model must be downloaded

4. After downloading the model, put it in the ControlNet plug-in folder that was installed before

5. Restart the interface

6. The corresponding model can be found

7. Try it yourself

4. Other ways to download and install ControlNet plug-ins

1. You can copy the git address of the ControlNet plug-in on Github, and then install it in the extension-install from the URL

ControlNet plugin GitHub address: https://github.com/Mikubill/sd-webui-controlnet

2. You can copy the git address of the ControlNet plug-in on Github, and then install it in git clone

Note: Clone in xxxx\xxxx\stable-diffusion-webui\extensions folder

3. You can download the ControlNet plug-in package from Github and unzip it to the xxxx\xxxx\stable-diffusion-webui\extensions folder