Learn Deep Learning Background
Recently, I am working on a ticket grabbing software. Since 12306 often checks the account status, I need to log in again when I grab a ticket, and then I need a verification code to log in. So I first thought of using java based on the perceptual hash algorithm pHash to do similarity matching and identification, but the test found that the accuracy was poor. Simple logic here. The 12306 verification code is generated and stitched together from 8 small pictures. Every time a request is made, a random image is drawn in the background of the server to make a verification code. The size of each thumbnail is the same. After cutting the picture using java, 8 small pictures are generated. The pHash algorithm is used to match image similarity. As a result, the recognition degree is very low, and there is no need to continue to do the following logic, such as training labels. Do matching classification pictures and so on. So I thought of the current image recognition classification in deep learning. Due to zero basic contact, I also borrowed a lot of related articles and PDFs before installation. After a few days of study and installation, Huangtian paid off the bumps and bumps along the way and finally installed caffe, so this article is more suitable for newcomers. It is better to teach a man how to fish than to teach a man how to fish. I will not post the compiled caffe. I summed up this tutorial just so that friends who are new to caffe like me can successfully install this NB thing.
1: Environmental preparation
Installation preface: This article is suitable for newcomers to read, so the installation steps will be a bit long-winded, and the second is that we use CPU for training, so there is no need to install Cuda and CuDNN.
1. Install git
Since the installation is recorded here using the Windows system, most operations still need to be performed using linux commands. So in order to facilitate the execution of linux commands, you need to git first.
Keep the default option and click next. Since I use it personally, I select the second item: Use Git from the Windows Command Prompt and then continue with the default option.
After the installation is complete, remember to check whether it is available to ensure that in case, some computers may have environment variables that are not automatically configured.
If the configured environment variables do not appear, please configure them yourself:
Find Path in system variables and add: C:\Program Files\Git\cmd;
2. Install Visual Studio 2015 (hereinafter referred to as VS2015)
Download address + permanent registration code
3. Install CMake & version>=3.4
3.1 Download address Cmake (version 3.11.1)
Download address: https://download.csdn.net/download/u014714937/10375108 or https://cmake.org/download/
After the download is complete, unzip it:
3.2 Configure environment variables
Also find Path in the system variable and append the path extracted by your cmake, my computer path is > D:\DeepLearning\cmake-3.11.1-win64-x64\bin
Note: If it is not used after the git environment variable; Remember to manually add ; at the end of the symbol before cmake appends ;
3.3 Test cmake
Open CDM > enter cmake -version If the version information appears, it means that the environment variable configuration is successful
C:\Users\Administrator>cmake -version cmake version 3.11.1 CMake suite maintained and supported by Kitware (kitware.com/cmake).
4. Install Python
4.1 Download python
Download address: https://download.csdn.net/download/u014714937/10375211
4.2 Install python
When installing, remember to check "Add Python 3.6 to PATH" to indicate that if there is no special requirement for the configuration of python environment variables, it is recommended to use the default installation: Install Now.
Normally it will be installed in about a minute.
4.3 Test python
4.4 Install other python related modules
4.4.1 Installing Anaconda
Download address: https://www.anaconda.com/download/
Download Python 3.6 version for Python 3.6, and Python 2.7 version for Python 2.7.
Select the user during installation. If there is no specific requirement, it is recommended to all users (All Users). After selecting the path, there is no specific requirement, and the installation can be started with the next step. The file is large and more than 2 GB are extracted after decompression. The installation process needs to wait patiently for a few minutes.
After the installation is complete, there will be a question about whether to install the Microsoft Express Development Kit. Install Microsoft VSCode Click to install, and click Skip after the installation is complete.
4.4.2 Install pip
Download address: https://pypi.org/project/pip/#files
After the download is complete, extract it to a folder. Here is a folder "dist" that may appear after decompression, and then there is a compressed package in this folder. If this happens, you need to decompress the real compressed package to the directory.
After the decompression is complete, hold down Shift + right mouse button in the current directory and click "Open command window here" or use the CMD console to enter the decompression directory and enter:
python setup.py install
installation failed:
If the installation command is entered, the reason for this is because the execution command cannot be found in the current directory. Confirm whether it is executed in the main directory of the decompressed PIP file.
D:\DeepLearning\dist>python setup.py install python: can't open file 'setup.py': [Errno 2] No such file or directory D:\DeepLearning\dist>
The installation is complete:
Detect installation:
Open CDM and enter pip list. If the version information appears, the installation is successful.
D:\DeepLearning\dist>pip list Package Version ---------- ------- pip 10.0.1 setuptools 28.8.0
4.4.3 Install six: pip install six
D:\DeepLearning\dist>pip install six Collecting six Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa 78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl Installing collected packages: six Successfully installed six-1.11.0
4.4.4 Install yaml: pip install pyyaml
D:\DeepLearning\dist>pip install pyyaml
Collecting pyyaml
Downloading https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0
eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz (253kB)
40% |█████████████ | 102kB 195kB/s eta 0:00:0
44% |██████████████▎ | 112kB 286kB/s eta 0:00
48% |███████████████▌ | 122kB 276kB/s eta 0:0
52% |████████████████▉ | 133kB 273kB/s eta 0:
56% |██████████████████▏ | 143kB 421kB/s eta
60% |███████████████████▍ | 153kB 354kB/s eta
64% |████████████████████▊ | 163kB 345kB/s et
68% |██████████████████████ | 174kB 382kB/s e
72% |███████████████████████▎ | 184kB 530kB/s
76% |████████████████████████▋ | 194kB 536kB/
80% |██████████████████████████ | 204kB 544kB
84% |███████████████████████████▏ | 215kB 522
89% |████████████████████████████▌ | 225kB 61
93% |█████████████████████████████▉ | 235kB 7
97% |███████████████████████████████ | 245kB
100% |████████████████████████████████| 256k
B 1.6MB/s
Installing collected packages: pyyaml
Running setup.py install for pyyaml ... done
Successfully installed pyyaml-3.12
4.4.5 Install numpy: pip install numpy
The installation process may occur in the following situations:
THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have u
pdated the package versions, please update the hashes. Otherwise, examine the pa
ckage contents carefully; someone may have tampered with them.
numpy from https://files.pythonhosted.org/packages/30/70/cd94a1655d082b8f024
b21af1eb13dd0f3035ffe78ff43d4ff9bb97baa5f/numpy-1.14.2-cp36-none-win_amd64.whl#s
ha256=6be6b0ca705321c178c9858e5ad5611af664bbdfae1df1541f938a840a103888:
Expected sha256 6be6b0ca705321c178c9858e5ad5611af664bbdfae1df1541f938a84
0a103888
Got 6e3df481acb4ee8249957a82771e7a23ab89368e7914bb131c368e83
b976f270
The main reason: due to the problem of network speed, the downloaded files are not completely mismatched file MD5
Solution: Re-execute the command to download the file and install it, and try several times.
The installation is complete:
Installing collected packages: numpy Successfully installed numpy-1.14.2
All the above things are ready to start.
5. Download caffe and compile
5.1 Download caffe
Small suggestion: We use the version provided by the caffe official website here, and some friends use the windows version provided by Microsoft. If you use the windows version provided by Microsoft, it seems that there is a windows folder in the caffe root directory, and the contents of the folder That is, the caffe-related VS files have been created.
Therefore, there may be differences in the installation and use process. It is recommended to use the version compiled and installed by yourself. Because using the caffe file that has been created and installed may encounter some unknown problems due to different environments. For example, if the VS file created in the downloaded caffe is created by VS2013 and the platform version (SKD) is different from the local installation, it will take more time to adjust the configuration.
Use the git command to download git clone https://github.com/BVLC/caffe.git You can also download and unzip it manually
$ git clone https://github.com/BVLC/caffe.git Cloning into 'caffe'... remote: Counting objects: 54253, done. remote: Total 54253 (delta 0), reused 0 (delta 0), pack-reused 54253 Receiving objects: 100% (54253/54253), 62.60 MiB | 359.00 KiB/s, done. Resolving deltas: 100% (36633/36633), done. Administrator@STI5X9B683AVXOY MINGW64 /d/DeepLearning/caffe $
The initial directory structure is as follows:
5.2 Modify the configuration and execute the creation of the caffe file (this step is very important)
Find and edit build_win.cmd through the caffe root directory and enter the scripts folder.
The main modified part of the statement is from line 7 to line 31. Between lines 70 and 99, the specific modification should be determined by the following steps.
Modify the if statement in lines 7-31 (the description of the modified items in if is the same as modifying the code logic between lines 70-99):
Modify lines between 70 and 99
By comparison, it can be found that the codes in if and else are similar, but the lines of code corresponding to the Anaconda version path in python are missing. The default file does not have these lines of code, we need to manually copy the modified code in the if to the else code, and the modification is as follows:
EDIT The main modified items are:
1. Modify to CPU computing training CPU_ONLY=1
2. Modify the corresponding VS version to generate code MSVC_VERSION=14 // The VS version of 2013 is modified to 12 The VS version of 2015 is modified to 14
3. Modify the compilation method, use VS to compile without using NinJa to compile WITH_NINJA=0 //It must be set to 0 here. Unless you want to compile with the ninja compiler.
4. Modify the corresponding python version PYTHON_VERSION=3 //The python version of 2.7 should be changed to 2, and the python version of 3.5 should be changed to 3
5. Modify the Anaconda version corresponding to python development //If the third modification PYTHON_VERSION is 3, the version of python2.7 will be installed according to the following modification, which will modify the record of EQU 2
if !PYTHON_VERSION! EQU 3 (
set CONDA_ROOT=Anaconda3 corresponds to the directory address of the installation
)
Note: The above modification with // description needs to be determined by the computer environment.
5.3 Create a caffe executable VS file
When the above content is modified, let's run build_win.cmd to create the VS executable file. It is the caffe file used for compilation. Shift + right mouse button in the scripts directory under the caffe root directory to open "Open command window here" and enter start build_win.cmd.
First time driving - (overturned):
D:\DeepLearning\caffe\caffe\scripts>start build_win.cmd
It seems to be unsuccessful after execution.
'"D:\VisualStudio\Common7\Tools\..\..\VC\vcvarsall.bat"' is not an internal or external command, nor not a runnable program or batch file. CMake Error at CMakeLists.txt:18 (project): Failed to run MSBuild command:....... 10,000 words omitted here
An exception has occurred, what is the error? It's another person who pays off. After various tests, it was found that it was because of the problem of the VS installation environment. Like when I installed this VS2015 before, because I didn't need to develop C++-related content, I chose a custom installation during installation and didn't check the C++-related compiler and "underpants".
Therefore, C++ related commands cannot be found when compiling the code. Found the problem and how to solve it? The method is very simple, just reinstall the VS development tools.
No, no, don't worry, there is another solution. Maybe you don't know that the VS installation tool is so powerful, you don't need to uninstall and reinstall it, it can be like patching, then reopen the installer and select the features you don't have installed to install. Check all Visual C++ in the programming language and click Next to install it. Generally, it takes about half an hour to install these items (depending on the performance of the computer).
Everything is ready and only owed to the east wind. After installing the C++ related components, we will execute the start build_win.cmd command again ( Note: Remember to clear the build folder under the build_win.cmd file directory before executing the build_win.cmd command each time ). ........
Second drive - (overturned):
The system cannot find the drive specified. The system cannot find the drive specified. 10,000 words are omitted here....... ===================================== ========================= -- The C compiler identification is MSVC 19.0.24215.1 -- The CXX compiler identification is MSVC 19.0.24215.1 -- Check for working C compiler: D:/VisualStudio/VC/bin/x86_amd64/cl.exe -- Check for working C compiler: D:/VisualStudio/VC/bin/x86_amd64/cl.exe -- work s -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: D:/VisualStudio/VC/bin/x86_amd64/cl.exe -- Check for working CXX compiler: D:/VisualStudio/VC/bin/x86_amd64/cl.exe -- wo rks -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found PythonInterp: D:/DeepLearning/Anaconda3/python.exe (found suitable vers ion "3.6.4", minimum required is "2.7") CMake Error at cmake/WindowsDownloadPrebuiltDependencies.cmake:40 (message): Could not find url for MSVC version = 1900 and Python version = 3.6. Call Stack (most recent call first): CMakeLists.txt:77 (include) -- Configuring incomplete, errors occurred! See also "D:/DeepLearning/caffe/caffe/scripts/build/CMakeFiles/CMakeOutput.log". ERROR: Configure failed D:\DeepLearning\caffe\caffe\scripts>
It seems to have failed again, and the exception description seems to be caused by inconsistent python versions. So I installed the Anaconda2 version again and modified the file build_win.cmd file
Match the corresponding python version to the corresponding version of Anaconda
:: Set python 2.7 with conda as the default python
if !PYTHON_VERSION! EQU 2 (
set CONDA_ROOT=D:\DeepLearning\Anaconda2
)
:: Set python 3.5 with conda as the default python
if !PYTHON_VERSION! EQU 3 (
set CONDA_ROOT=D:\DeepLearning\Anaconda3
)
3rd drive - (overturned):
-- Found PythonInterp: D:/DeepLearning/Anaconda3/python.exe (found suitable vers ion "3.6.4", minimum required is "2.7") CMake Error at cmake/WindowsDownloadPrebuiltDependencies.cmake:40 (message): Could not find url for MSVC version = 1900 and Python version = 3.6. Call Stack (most recent call first):
When I overturned the car for the third time, I was thinking, am I not suitable for this awesome thing, but there is no way, since I got on the high speed, how can I get off the high speed casually. So I modified it again, and modified the python version in the build_win.cmd file to match the corresponding Anaconda to match Anaconda2. So....
Fourth drive - (flying up):
Another king who lives up to his heart. It worked.
First look at the modified configuration file (remember that if and else have to be modified), as for why this is the case, I am also a hen. . . It may be caused by my configuration or environment. Logically, what I configure is that the Anaconda configuration of python3.6 should also be version 3, but I can't pass the reference to Anaconda3, so I can only change it to Anaconda2 and let it pass first, and then I see when I read the information. Some bosses said that if the installation of Anaconda2/3 here fails, you need to install Miniconda2. The original meaning is that the following references are all Miniconda needs to install this. But my local test here is normal, so I have not installed it. If you need to install it, download and install it here. The installation interface is the same as when installing Anaconda. Download address: https://conda.io/miniconda.html :
:: Set python 2.7 with conda as the default python
if !PYTHON_VERSION! EQU 2 (
set CONDA_ROOT=D:\DeepLearning\Anaconda2
)
:: Set python 3.5 with conda as the default python
if !PYTHON_VERSION! EQU 3 (
set CONDA_ROOT=D:\DeepLearning\Anaconda2
)
Then you will find that the console is silently downloading a file, and the downloaded content is the dependencies required to generate the caffe project. After the download is complete, the sln file of the corresponding version of Visual Studio is automatically generated. So we can wait patiently. ........When the dependencies are downloaded, the project will be generated.
After waiting patiently for a few minutes, the generation is completed.
Don't pay attention to the warning, we go directly to the generated folder to open the caffe project and compile it. It is the build folder in the build_win.cmd file directory.
After entering this folder, the results are roughly the same, the versions are different, and the generated files are slightly different. Then double-click the Caffe.sln file to open it automatically with VS.
5.4 Compile caffe
When I translated the materials, I found that many big guys said that they used the Release method when generating solutions. Then the platform version is adjusted to X64. I will not talk about the difference between using Release and Debug here. But the compilation mode here will not directly affect the use of caffe, so Debug is also possible.
After waiting for the VS tool to load the project, we click Generate > Generate Solution in the menu bar, which is also possible through the project bar on the right. The result is the same. If the compilation passes normally, the console prints the following information:
1001 words are omitted here 41> -- Creating hardlink for opencv_imgcodecs310d.dll in D:/DeepLearning/caffe/caffe/scripts/build/test/Debug 41> -- Failed to create hardlink D:/DeepLearning/caffe/caffe/scripts/build/test/Debug/opencv_imgcodecs310d.dll. Copying instead. 41> -- Copying opencv_imgcodecs310d.dll to D:/DeepLearning/caffe/caffe/scripts/build/test/Debug 41> -- Creating hardlink for opencv_imgproc310d.dll in D:/DeepLearning/caffe/caffe/scripts/build/test/Debug 41> -- Failed to create hardlink D:/DeepLearning/caffe/caffe/scripts/build/test/Debug/opencv_imgproc310d.dll. Copying instead. 41> -- Copying opencv_imgproc310d.dll to D:/DeepLearning/caffe/caffe/scripts/build/test/Debug 41> -- Creating hardlink for python27.dll in D:/DeepLearning/caffe/caffe/scripts/build/test/Debug 45>------ Build skipped: Project: runtest, Configuration: Debug x64 ------ 45>The project to build is not selected for this solution configuration ========== Generated: 40 successful, 0 failed, 0 latest, 5 skipped ==========