Windows on Snapdragon User Guide (5)

---

4 Qualcomm® Neural Processing SDK for Artificial Intelligence

The Qualcomm® Neural Processing SDK for AI simplifies the process for developers to run optimized neural networks on Snapdragon devices without the need for a cloud connection. Qualcomm Neural Processing SDK leverages Qualcomm® AI Engine direct support for each accelerator backend.
As part of the Qualcomm® Artificial Intelligence Stack, the SDK facilitates the integration of neural networks on devices. It supports multiple trained neural network models in PyTorch , ONNX , or TensorFlow / TensorFlow Lite , and can run on Qualcomm® Snapdragon CPU or DSP processors. The SDK includes tools for model conversion and execution, APIs for core positioning, and support for convolutional neural networks and custom operators. By offloading the burden of running neural networks on Qualcomm® Snapdragon devices, the Qualcomm® Neural Processing SDK allows developers to focus on creating new and innovative user experiences.

Key Features
This SDK is a solution for developers who want to perform AI/ML inference directly on edge devices. By leveraging the computing power of the Qualcomm® Snapdragon Mobile Platform, the Qualcomm® Neural Processing SDK enables on-device processing, providing the following key benefits:

• Reduced latency : With on-device processing, the time and latency associated with sending data to the cloud and receiving results is eliminated, making it ideal for real-time applications.
• Improved privacy : By keeping computing local, user data remains on the device and is not transferred to the cloud, providing a higher level of privacy and security.
• Cost savings : On-device processing eliminates the need for cloud connectivity, which can reduce the cost of deploying AI/ML applications, especially in areas where cloud connectivity is limited or expensive.
• Improved reliability : By reducing reliance on the cloud, on-device processing provides a more reliable solution as it is less susceptible to network outages or connectivity issues.
• Optimized inference performance : The SDK helps developers optimize the performance of AI/ML workloads on Qualcomm Snapdragon platforms by:
  • Heterogeneous Computing : The SDK leverages the powerful computing power of the Snapdragon platform, including CPU and DSP, to provide optimal performance for artificial intelligence/machine learning workloads.
  • Optimized model conversion : Developers can convert deep learning models trained in popular frameworks such as PyTorch , ONNX , TensorFlow , and TensorFlow Lite into formats optimized for Qualcomm® Snapdragon platforms. This helps improve model performance and efficiency during inference.
  • Low-precision calibration : The SDK supports low-precision inference, which reduces the number of bits used to represent model parameters during inference. This improves performance while maintaining high accuracy, giving developers the flexibility to trade off accuracy and performance as needed.
  • Custom operator support : The SDK provides an API to implement custom layers, allowing developers to extend the functionality of their models to meet the specific needs of their applications.
  • Efficient execution : The SDK provides APIs targeting Qualcomm® Snapdragon cores that best match the desired user experience, allowing developers to achieve optimal performance for their AI/ML workloads.

Develop AI applications using the Qualcomm® Neural Processing SDK
To build and run AI applications for Windows on Snapdragon (WoS) or Linux, use the Qualcomm® Neural Processing SDK to generate a Deep Learning Container (DLC).

Setting up the Neural Processing SDK
The Qualcomm Neural Processing SDK needs to be installed on the host machine (x86 64-bit machine).

Build Artificial Intelligence Applications
Build applications for Windows or Linux platforms.

Run Windows application on target device

4.1 Develop artificial intelligence applications using Qualcomm® Neural Processing SDK

To build and run AI applications for Windows on Snapdragon (WoS) or Linux, use the Qualcomm® Neural Processing SDK to generate a Deep Learning Container (DLC).

recommend	resource
See the Qualcomm® Neural Processing Engine SDK documentation to learn how to convert ONNX, PyTorch, TensorFlow and TensorFlow Lite models to DLC files, quantize DLC to 8-bit or 16-bit fixed point to run on Qualcomm® Hexagon™ DSP, analyze performance and Integrate networking into applications.	Qualcomm® Neural Processing Engine SDK Qualcomm® Neural Processing Engine SDK Reference Manual

4.1.1 Workstation requirements

• x86_64 Linux workstation for DLC preparation, generation and quantification
  • Ubuntu 20.04
  • Python 3.8
  • One of the following frameworks
    • ONNX
    • PyTorch
    • TensorFlow
    • TensorFlow Lite

4.1.2 Device requirements to support Qualcomm® Neural Processing SDK

• Windows ARM64 device builds the application and runs the compiled model binary
  • Windows 11
  • Visual Studio 2022 17.6.4
    • Windows 11 SDK (10.0.22621.0)
    • MSBuild support for LLVM (clang-cl) toolset
    • C++ Clang Compiler for Windows (15.0.1)
    • MSVC v143 – VS 2022 C++ ARM64/ARM64EC Build Tools – (Latest)
    • C++ CMake tools for Windows

4.2 Set up neural processing SDK

Qualcomm Neural Processing SDK needs to be installed on the host machine (x86 64-bit machine). Before generating the DLC, set up the Qualcomm® Neural Processing SDK
on an Ubuntu 20.04 Linux machine .

1. Enable Windows Subsystem for Linux from the Windows Control Panel.
   a. In Control Panel , search for Turn Windows features on or off .
   b. In the Windows Features dialog box, select Windows Subsystem for Linux . Click OK .
   
   c. Click Restart now to restart the computer.
2. Choose one of the following options to install WSL or native Linux Ubuntu 20.04.6 on an x86_64 device. This enables the SDK to be used with the ONNX framework.
   • In the Microsoft Store, click Get to install Ubuntu 20.04 WSL.
     
   • Click Open and create an ID and password when prompted.
     

Installing, this may take a few minutes...
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username: %UserName%
New password:
Retype new password:
passwd: password updated successfully
Installation successful!
Windows Subsystem for Linux is now available in the Microsoft Store!
You can upgrade by running 'wsl.exe --update' or by visiting https://aka.ms/wslstorepage
Installing WSL from the Microsoft Store will give you the latest WSL updates, faster.
For more information please visit https://aka.ms/wslstoreinfo

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 4.4.0-19041-Microsoft x86_64)

3. Install the latest version of the Qualcomm® Neural Processing SDK from the Qualcomm® Package Manager .

NOTE You will be prompted to install Qualcomm® Package Manager 3
if it is not already installed .

The Neural Processing SDK will be installed in the C:\Qualcomm\AIStack\SNPE<build_version> folder. Where <build_version> corresponds to the specific neural processing SDK release version. For example, C:\Qualcomm\AIStack\SNPE\2.13.0.230730.

Note
The installation path will be different for each SDK version. The above paths and the pictures below are for reference only and do not represent the SDK version installed on your machine.

4. Copy the installed SDK to the \wsl folder on the X86_64 host device.
   

Note that
the x86_64-linux-clang binaries are intended for use with WSL or X86_64 host devices for Linux. It does not work with WSL on ARM64 devices.

If you encounter snpe-sdk permission issues, use the following command to grant permissions:

sudo chown -R $USER:$USER snpe-sdk && sudo chmod -R 777 snpe-sdk

a. Complete the settings of SDK dependencies, use: Qualcomm Neural Processing Engine SDK .
b. For the remainder of this guide, it is important to install the SDK dependencies, DNN framework, and other requirements as described in the SDK documentation above.

5. Validate the SNPE toolchain on host x86_64 by using ML models for model preparation, transformation and quantification.
   The Qualcomm® AI Neural Processing SDK does not bundle any model files, but does include scripts to download and convert certain models to the Qualcomm Deep Learning Container (.dlc) format.

   a. Prepare pre-trained model and input files. (Tested using Resnet50 Onnx model.)

   wget --no-check-certificate -c https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v1-7.onnx 
   
   python3 ~/snpe-sdk/examples/Models/inception_v3/scripts/create_inceptionv3_raws.py -i ~/snpe-sdk/examples/Models/inception_v3/data -d . -s 224
                   
   python3 ~/snpe-sdk/examples/Models/inception_v3/scripts/create_file_list.py -e "*.raw"
   
   

   b. Run the following command to convert the pretrained Resnet50 example to DLC format:

   snpe-onnx-to-dlc -i resnet50-v1-7.onnx --input_dim data 1,3,224,224
   

   For example, a message similar to the following will be displayed.

   ce@dev:~$ snpe-onnx-to-dlc -i resnet50-v1-7.onnx --input_dim data 1,3,224,224
   WARNING: The argument 'input_shapes' is deprecated. Please use 'overwrite_input_shapes' and/or
   'test_input_shapes' instead. An error will be raised in the future.
   2023-04-25 00:29:48,893 - 214 - INFO - Successfully simplified the onnx model in child process
   2023-04-25 00:31:56,557 - 214 - INFO - Successfully receive the simplified onnx model in main process
   2023-04-25 00:34:07,156 - 219 - WARNING - WARNING_GEMM: GEMM operation is not supported in the general case, 
    attempting to interpret as FC
   2023-04-25 00:34:07,696 - 214 - INFO - INFO_INITIALIZATION_SUCCESS:
   2023-04-25 00:34:08,464 - 214 - INFO - INFO_CONVERSION_SUCCESS: Conversion completed successfully
   2023-04-25 00:34:09,320 - 214 - INFO - INFO_WRITE_SUCCESS: 
   

   c. Quantize DSP/HTP graphics and add them to DLC for runtime DSP. The default SoC value is SM8550. Use --htp_socs to match the HTP SoC version on the target device

   snpe-dlc-quantize --input_dlc=resnet50-v1-7.dlc --input_list=file_list.txt --enable_htp --htp_socs=sm8350 --silent
   

   For example, a message similar to the following will be displayed.

   [WARN] verbose coredump is not supported
   [ERROR] Failed to enable verbose core dump.
   [INFO] DebugLog shutting down.
        1.3ms [  INFO ] Initializing logging in the backend. Callback: [0xc42410], Log Level: [3]
        1.9ms [  INFO ] No BackendExtensions lib provided;initializing NetRunBackend Interface
      472.7ms [  INFO ] cleaning up resources for input tensors
      473.2ms [  INFO ] cleaning up resources for output tensors
      677.5ms [  INFO ] cleaning up resources for input tensors
      678.2ms [  INFO ] cleaning up resources for output tensors
      877.9ms [  INFO ] cleaning up resources for input tensors
      878.6ms [  INFO ] cleaning up resources for output tensors
     1075.2ms [  INFO ] cleaning up resources for input tensors
     1075.9ms [  INFO ] cleaning up resources for output tensors
   [WARN] verbose coredump is not supported
   [ERROR] Failed to enable verbose core dump.
   [INFO] DebugLog shutting down.
   

   d. Generate the output (OP_NAME.raw) file under ./output/Result_0/ when the CPU is running.

   snpe-net-run --container resnet50-v1-7_quantized.dlc --input_list file_list.txt
   

   For example, a message similar to the following will be displayed.

   -------------------------------------------------------------------------------
   Model String: N/A
   SNPE v2.12.1.230626174329_59328
   -------------------------------------------------------------------------------
   Processing DNN input(s):
   /home/%UserName%/model/chairs.raw
   Processing DNN input(s):
   /home/%UserName%/model/notice_sign.raw
   Processing DNN input(s):
   /home/%UserName%/model/plastic_cup.raw
   Processing DNN input(s):
   /home/%UserName%/model/trash_bin.raw
   Successfully executed!