Algorithm on Kang's Blog

Ascend NPU Deployment Guide for Kubernetes

Thu, 08 Jan 2026 00:00:00 +0000

Ascend NPU Deployment Guide for Kubernetes

Overview

This document describes the complete process of deploying the Ascend NPU containerized environment in a Kubernetes cluster, suitable for the following scenarios:

Container Runtime: Containerd
NPU Device: Ascend 310P
System Architecture: aarch64 (ARM64)
Kubernetes Version: 1.28+

The deployment process includes three main steps:

Environment preparation (node labels, users, directories)
Install Ascend Docker Runtime
Deploy Ascend Device Plugin

Preparation

Create Node Labels

Add appropriate labels to Kubernetes nodes for subsequent Pod scheduling and resource management.

# Label master node
kubectl label nodes ecs-b0tf90001 masterselector=dls-master-node

# Label NPU compute nodes
kubectl label nodes ecs-exyqec0002 node-role.kubernetes.io/worker=worker
kubectl label nodes ecs-exyqec0002 workerselector=dls-worker-node
kubectl label nodes ecs-exyqec0002 host-arch=huawei-arm
kubectl label nodes ecs-exyqec0002 accelerator=huawei-Ascend310P

Note: Please modify ecs-b0tf90001 and ecs-exyqec0002 according to your actual node names.

Create System User

Create dedicated users and groups on Ascend compute nodes (e.g., ecs-exyqec0002).

# Create hwMindX user (UID 9000)
useradd -d /home/hwMindX -u 9000 -m -s /sbin/nologin hwMindX

# Create HwHiAiUser group
groupadd HwHiAiUser

# Add hwMindX user to HwHiAiUser group
usermod -a -G HwHiAiUser hwMindX

Important: UID 9000 and group HwHiAiUser are the default configurations for the Ascend software stack. Do not modify them arbitrarily.

Create Log Directory

Create the Device Plugin log directory on Ascend compute nodes.

1
2
3

# Create log directory and set permissions
mkdir -m 750 /var/log/mindx-dl/devicePlugin
chown root:root /var/log/mindx-dl/devicePlugin

Install Ascend Docker Runtime

Ascend Docker Runtime is a core component for using Ascend NPU in containerized environments and must be installed on all Ascend compute nodes.

Download Installation Package

Visit the official Git repository to download the corresponding version of the installation package.

Example: Ascend-docker-runtime_7.2.RC1.SPC2_linux-aarch64.run

Installation Steps

Step 1: Enter Installation Package Directory

`1`	`cd <path to run package>`

Step 2: Verify Package Integrity

`1`	`./Ascend-docker-runtime_{version}_linux-{arch}.run --check`

Expected output:

1
2

[WARNING]: --check is meaningless for Ascend-docker-runtime and will be discarded in the future
Verifying archive integrity... All good.

Step 3: Add Executable Permission

`1`	`chmod u+x Ascend-docker-runtime_{version}_linux-{arch}.run`

Step 4: Execute Installation

Method 1: Install to Default Path (Recommended)

`1`	`./Ascend-docker-runtime_{version}_linux-{arch}.run --install`

Method 2: Install to Custom Path

`1`	`./Ascend-docker-runtime_{version}_linux-{arch}.run --install --install-path=<path>`

Successful installation output example:

Uncompressing ascend-docker-runtime  100%
[INFO]: installing ascend docker runtime
...
[INFO] Ascend Docker Runtime install success

Default Installation Path: /usr/local/Ascend/Ascend-Docker-Runtime/

Configure Containerd

Modify the Containerd configuration file /etc/containerd/config.toml according to the cgroup version used by your system.

Configuration Method 1: Cgroup v1

Two key configuration items need to be modified:

runtime_type = "io.containerd.runtime.v1.linux"
runtime = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"

Complete configuration example:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runtime.v1.linux"
    runtime_engine = ""
    runtime_root = ""
    privileged_without_host_devices = false
    base_runtime_spec = ""
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]

[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/opt/cni/bin"
  conf_dir = "/etc/cni/net.d"
  max_conf_num = 1
  conf_template = ""

[plugins."io.containerd.grpc.v1.cri".registry]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["https://registry-1.docker.io"]

[plugins."io.containerd.grpc.v1.cri".image_decryption]
  key_model = ""

# ... other configurations ...

[plugins."io.containerd.monitor.v1.cgroups"]
  no_prometheus = false

[plugins."io.containerd.runtime.v1.linux"]
  shim = "containerd-shim"
  runtime = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"
  runtime_root = ""
  no_shim = false
  shim_debug = false

[plugins."io.containerd.runtime.v2.task"]
  platforms = ["linux/amd64"]

Configuration Method 2: Cgroup v2

The following key configuration item needs to be modified:

BinaryName = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"

Complete configuration example:

[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
  [plugins."io.containerd.grpc.v2.cri".containerd.runtimes.runc]
    base_runtime_spec = ""
    cni_conf_dir = ""
    cni_max_conf_num = 0
    container_annotations = []
    pod_annotations = []
    privileged_without_host_devices = false
    runtime_engine = ""
    runtime_path = ""
    runtime_root = ""
    runtime_type = "io.containerd.runc.v2"
    
    [plugins."io.containerd.grpc.v2.cri".containerd.runtimes.runc.options]
      BinaryName = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"
      CriuImagePath = ""
      CriuPath = ""
      CriuWorkPath = ""
      IoGid = 0
      IoUid = 0
      NoNewKeyring = false
      NoPivotRoot = false
      Root = ""
      ShimCgroup = ""
      SystemdCgroup = true

Tip: To check the cgroup version, execute stat -fc %T /sys/fs/cgroup/. Output cgroup2fs indicates v2, while tmpfs indicates v1.

Restart Services

1
2

systemctl daemon-reload
systemctl restart containerd kubelet

Verify Installation

Execute the following command on the Kubernetes master node to confirm that the Ascend compute nodes are in normal status:

`1`	`kubectl get nodes`

Expected output example:

1
2
3

NAME              STATUS   ROLES           AGE   VERSION
k8s-master        Ready    master,worker   3d    v1.28.12
k8s-worker        Ready    worker          3d    v1.28.12

All nodes should show Ready status.

Deploy Ascend Device Plugin

Ascend Device Plugin is responsible for managing and allocating NPU resources in Kubernetes.

Prepare Images

1. Pull Images

Execute the following commands on Ascend compute nodes to pull the required images:

# Pull all required images
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/resilience-controller:v7.1.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-operator:v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/npu-exporter:v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-k8sdeviceplugin:v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-controller-manager:v1.7.0-v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-scheduler:v1.7.0-v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/noded:v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/clusterd:v7.2.RC1

2. Export Images

docker save -o ascend.tar \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/resilience-controller:v7.1.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-operator:v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/npu-exporter:v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-k8sdeviceplugin:v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-controller-manager:v1.7.0-v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-scheduler:v1.7.0-v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/noded:v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/clusterd:v7.2.RC1

3. Import to Containerd

`1`	`ctr -n k8s.io images import ascend.tar`

Download Deployment Configuration Files

Visit the official Git repository to download the Ascend Device Plugin installation package.

Example: Ascend-mindxdl-device-plugin_7.2.RC1.SPC2_linux-aarch64.zip

After extracting, copy the corresponding YAML files to the Kubernetes management node.

Choose the Appropriate YAML File

Select the corresponding YAML file based on the actual device type and whether to use the Volcano scheduler:

YAML Filename	Use Case
`device-plugin-310-v{version}.yaml`	Atlas 300I inference card, without Volcano
`device-plugin-310-volcano-v{version}.yaml`	Atlas 300I inference card, with Volcano
`device-plugin-310P-1usoc-v{version}.yaml`	Atlas 200I SoC A1 core board, without Volcano
`device-plugin-310P-1usoc-volcano-v{version}.yaml`	Atlas 200I SoC A1 core board, with Volcano
`device-plugin-310P-v{version}.yaml`	Atlas inference series products (e.g., 310P), without Volcano
`device-plugin-310P-volcano-v{version}.yaml`	Atlas inference series products, with Volcano
`device-plugin-910-v{version}.yaml`	Atlas training series products/A2/A3/800I A2, without Volcano
`device-plugin-volcano-v{version}.yaml`	Atlas training series products/A2/A3/800I A2, with Volcano

Note:

For Ascend 310P devices, typically choose device-plugin-310P-v{version}.yaml

Do not modify the DaemonSet.metadata.name field in the YAML file to avoid automatic identification issues

Deploy Device Plugin

1
2

# Please modify the image name to: swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-k8sdeviceplugin:v7.2.RC1
kubectl apply -f device-plugin-310P-v7.2.RC1.SPC2.yaml

Expected output:

serviceaccount/ascend-device-plugin-sa created
clusterrole.rbac.authorization.k8s.io/pods-node-ascend-device-plugin-role created
clusterrolebinding.rbac.authorization.k8s.io/pods-node-ascend-device-plugin-rolebinding created
daemonset.apps/ascend-device-plugin-daemonset created

Verify Deployment

Check if the Device Plugin started successfully:

`1`	`kubectl get pod -n kube-system \| grep ascend`

Expected output (status should be Running):

1
2

NAME                                   READY   STATUS    RESTARTS   AGE
ascend-device-plugin-daemonset-d5ctz   1/1     Running   0          11s

Check node NPU resources:

`1`	`kubectl describe node <node-name> \| grep -A 5 "Capacity:"`

You should see the huawei.com/Ascend310P resource.

Using NPU Compute Cards

Pod Configuration Example

Request NPU resources through the resources field in the Pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: npu-test-pod
spec:
  containers:
    - name: alg-container
      image: ubuntu:22.04
      resources:
        limits:
          memory: 24Gi
          huawei.com/Ascend310P: 1  # Request 1 NPU card
        requests:
          memory: 2Gi
          huawei.com/Ascend310P: 1  # Request 1 NPU card
      command: ["/bin/bash", "-c", "sleep infinity"]

Note:

The value of huawei.com/Ascend310P indicates the number of NPU cards requested

Values in limits and requests should be consistent

Adjust the resource name according to the actual NPU model (e.g., 310, 910, etc.)

NVIDIA GPU Driver Persistence

Fri, 07 Nov 2025 00:00:00 +0000

NVIDIA GPU Driver Persistence Configuration and Troubleshooting

Overview

This article documents a monitoring anomaly issue caused by GPU driver non-persistent mode, and introduces the principles and configuration methods of NVIDIA GPU driver persistence.

Problem Symptoms

During algorithm program stress testing, the Grafana monitoring dashboard revealed that the Nvidia Exporter service was running unstably, showing intermittent behavior:

Initial Investigation

Ruling Out Prometheus Scrape Issues
Manually executing the curl http://localhost:9835/metrics command on the target GPU server resulted in a timeout, confirming that the issue was with the Exporter service itself.

Adjusting Log Level
The Nvidia Exporter log level was adjusted to debug, but no obvious error messages were found.

Root Cause Identification
Manually executing the query command used internally by Nvidia Exporter:

nvidia-smi --query-gpu=timestamp,driver_version,vgpu_driver_capability.heterogenous_multivGPU,count,name,serial,uuid,pci.bus_id,pci.domain,pci.bus,pci.device,pci.baseClass,pci.subClass,pci.device_id,pci.sub_device_id,vgpu_device_capability.fractional_multiVgpu,vgpu_device_capability.heterogeneous_timeSlice_profile,vgpu_device_capability.heterogeneous_timeSlice_sizes,vgpu_device_capability.homogeneous_placements,pcie.link.gen.current,pcie.link.gen.gpucurrent,pcie.link.gen.max,pcie.link.gen.gpumax,pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max,index,display_mode,display_active,persistence_mode,addressing_mode,accounting.mode,accounting.buffer_size,driver_model.current,driver_model.pending,vbios_version,inforom.img,inforom.oem,inforom.ecc,inforom.pwr,gpu_recovery_action,gom.current,gom.pending,fan.speed,pstate,clocks_event_reasons.supported,clocks_event_reasons.active,clocks_event_reasons.gpu_idle,clocks_event_reasons.applications_clocks_setting,clocks_event_reasons.sw_power_cap,clocks_event_reasons.hw_slowdown,clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.hw_power_brake_slowdown,clocks_event_reasons.sw_thermal_slowdown,clocks_event_reasons.sync_boost,memory.total,memory.reserved,memory.used,memory.free,compute_mode,compute_cap,utilization.gpu,utilization.memory,utilization.encoder,utilization.decoder,utilization.jpeg,utilization.ofa,encoder.stats.sessionCount,encoder.stats.averageFps,encoder.stats.averageLatency,dramEncryption.mode.current,dramEncryption.mode.pending,ecc.mode.current,ecc.mode.pending,ecc.errors.corrected.volatile.device_memory,ecc.errors.corrected.volatile.dram,ecc.errors.corrected.volatile.register_file,ecc.errors.corrected.volatile.l1_cache,ecc.errors.corrected.volatile.l2_cache,ecc.errors.corrected.volatile.texture_memory,ecc.errors.corrected.volatile.cbu,ecc.errors.corrected.volatile.sram,ecc.errors.corrected.volatile.total,ecc.errors.corrected.aggregate.device_memory,ecc.errors.corrected.aggregate.dram,ecc.errors.corrected.aggregate.register_file,ecc.errors.corrected.aggregate.l1_cache,ecc.errors.corrected.aggregate.l2_cache,ecc.errors.corrected.aggregate.texture_memory,ecc.errors.corrected.aggregate.cbu,ecc.errors.corrected.aggregate.sram,ecc.errors.corrected.aggregate.total,ecc.errors.uncorrected.volatile.device_memory,ecc.errors.uncorrected.volatile.dram,ecc.errors.uncorrected.volatile.register_file,ecc.errors.uncorrected.volatile.l1_cache,ecc.errors.uncorrected.volatile.l2_cache,ecc.errors.uncorrected.volatile.texture_memory,ecc.errors.uncorrected.volatile.cbu,ecc.errors.uncorrected.volatile.sram,ecc.errors.uncorrected.volatile.total,ecc.errors.uncorrected.aggregate.device_memory,ecc.errors.uncorrected.aggregate.dram,ecc.errors.uncorrected.aggregate.register_file,ecc.errors.uncorrected.aggregate.l1_cache,ecc.errors.uncorrected.aggregate.l2_cache,ecc.errors.uncorrected.aggregate.texture_memory,ecc.errors.uncorrected.aggregate.cbu,ecc.errors.uncorrected.aggregate.sram,ecc.errors.uncorrected.aggregate.total,ecc.errors.uncorrected.volatile.sram.parity,ecc.errors.uncorrected.volatile.sram.secded,ecc.errors.uncorrected.aggregate.sram.parity,ecc.errors.uncorrected.aggregate.sram.secded,ecc.errors.uncorrected.aggregate.sram.thresholdExceeded,ecc.errors.uncorrected.aggregate.sram.l2,ecc.errors.uncorrected.aggregate.sram.sm,ecc.errors.uncorrected.aggregate.sram.mcu,ecc.errors.uncorrected.aggregate.sram.pcie,ecc.errors.uncorrected.aggregate.sram.other,retired_pages.single_bit_ecc.count,retired_pages.double_bit.count,retired_pages.pending,remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure,remapped_rows.histogram.max,remapped_rows.histogram.high,remapped_rows.histogram.partial,remapped_rows.histogram.low,remapped_rows.histogram.none,temperature.gpu,temperature.gpu.tlimit,temperature.memory,power.management,power.draw,power.draw.average,power.draw.instant,power.limit,enforced.power.limit,power.default_limit,power.min_limit,power.max_limit,module.power.draw.average,module.power.draw.instant,module.power.limit,module.enforced.power.limit,module.power.default_limit,module.power.min_limit,module.power.max_limit,clocks.current.graphics,clocks.current.sm,clocks.current.memory,clocks.current.video,clocks.applications.graphics,clocks.applications.memory,clocks.default_applications.graphics,clocks.default_applications.memory,clocks.max.graphics,clocks.max.sm,clocks.max.memory,mig.mode.current,mig.mode.pending,gsp.mode.current,gsp.mode.default,c2c.mode,protected_memory.total,protected_memory.used,protected_memory.free,fabric.state,fabric.status,platform.chassis_serial_number,platform.slot_number,platform.tray_index,platform.host_id,platform.peer_type,platform.module_id,platform.gpu_fabric_guid --format=csv

Key Finding: The command execution time fluctuated between 3-10 seconds, which was clearly abnormal. The test environment had 8 GPUs in total, with 2 being occupied by the algorithm program and the remaining 6 idle.

After reviewing the NVIDIA official documentation on GPU driver persistence, we attempted to enable persistent mode.

Solution

Temporarily Enable Persistent Mode

Execute the following command to immediately enable GPU driver persistence:

`1`	`nvidia-smi -pm 1`

After executing the query command again, the response time dropped to milliseconds, problem solved.

Configure Automatic Startup

To ensure the persistence configuration takes effect after system reboot, a systemd service needs to be configured:

1. Create Service Configuration File

`1`	`sudo vim /usr/lib/systemd/system/nvidia-persistenced.service`

2. Add the Following Content

[Unit]
Description=NVIDIA Persistence Daemon
Wants=network.target

[Service]
Type=forking
PIDFile=/var/run/nvidia-persistenced/nvidia-persistenced.pid
ExecStart=/usr/bin/nvidia-persistenced --persistence-mode
ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced

[Install]
WantedBy=multi-user.target

3. Enable and Start the Service

`1`	`sudo systemctl enable nvidia-persistenced.service --now`

Verification

After configuration, the monitoring system returned to normal, with stable GPU usage collection:

Technical Principles

GPU Driver Loading Mechanism

NVIDIA GPU interaction depends on the kernel mode driver, which operates in two modes:

Persistent Mode: The driver remains continuously active
On-Demand Loading Mode: The driver loads only when a program uses the GPU

Driver Lifecycle

Initialization Phase
When the first program attempts to interact with the GPU, if the kernel driver is not running, the system triggers driver loading and GPU device initialization.

De-initialization Phase
After all GPU client programs exit, the driver executes GPU de-initialization operations, essentially “shutting down” the GPU device.

Impact on Users

Application Startup Delay

When GPU initialization is triggered for the first time, operations such as ECC memory checks cause a delay of 1-3 seconds. If the GPU is already initialized, there is no such delay.

Driver State Loss

After GPU de-initialization, non-persistent state information (such as power limits, clock frequency configurations, etc.) is lost and restored to default values upon the next initialization. Enabling persistent mode avoids this issue.

Platform Differences

Windows Platform

On Windows systems, the kernel driver loads at system startup and remains running until system shutdown. Therefore, Windows users typically do not need to be concerned about driver persistence issues.

Note: Driver reload events (such as TDR triggers or driver updates) will cause non-persistent state resets.

Linux Platform

Linux system behavior depends on the runtime environment:

Graphical Environment
If the X Server runs on the target GPU, the kernel driver typically remains active from boot to shutdown, maintained by the X process connection.

Headless Server Environment
On servers without a graphical interface (Headless Server), if there is no long-running GPU client, each application start and stop will trigger driver loading and unloading. This is extremely common in High-Performance Computing (HPC) and Data Center environments, which was the root cause of this incident.

Best Practice Recommendations

Strongly recommended for production environments to enable GPU driver persistence, especially in headless server scenarios
Use systemd service to ensure persistence configuration automatically takes effect after system reboot
The monitoring system should be thoroughly tested after enabling persistence to verify the stability of metric collection
Regularly check the nvidia-persistenced service status to ensure it is running properly

References

NVIDIA Driver Persistence Official Documentation

Using PyInstaller to package PaddleOCR on Ascend 310P environment

Sun, 10 Aug 2025 00:00:00 +0000

Based on PaddlePaddle deployment on Ascend 310P, use PyInstaller to package PaddleOCR.

Scope and prerequisites

Target: Package a PaddleOCR-based CLI program for Ascend 310P NPU runtime.
OS/Arch: Linux x86_64 (example paths use Conda + Python 3.11).
Key versions:
- Python: 3.11 (adjust paths if you use a different version)
- PyInstaller: 6.x or newer
- PaddlePaddle (Ascend build) and runtime for 310P is correctly installed
Tip: Verify environment

1
2
3

python -V
pyinstaller --version
python -c "import paddle, cv2; print(paddle.__version__, cv2.__version__)"

Spec file

Package with the following test_ocr.spec:

# -*- mode: python ; coding: utf-8 -*-
from PyInstaller.utils.hooks import copy_metadata, collect_data_files

datas = [('/opt/PaddleX/paddlex', 'paddlex')] # Python package directory of the PaddleX project source code
datas += [('/root/miniconda3/envs/asr_ocr_npu/lib/python3.11/site-packages/paddle', 'paddle')] # Directory of the Paddle PyPI package
datas += collect_data_files('lmdb')
datas += copy_metadata('ftfy')
datas += copy_metadata('lxml')
datas += copy_metadata('opencv-contrib-python')
datas += copy_metadata('pyclipper')
datas += copy_metadata('pypdfium2')
datas += copy_metadata('ultra-infer-npu-python')
datas += copy_metadata('scikit-learn')


a = Analysis(
    ['./test_ocr.py'], # Entry script to be packaged
    pathex=[],
    binaries=[('/root/miniconda3/envs/asr_ocr_npu/lib/python3.11/site-packages/paddle/libs', '.')],  # Directory of Paddle shared libraries
    datas=datas,
    hiddenimports=['cv2', 'pypdfium2', 'ultra_infer', 'paddle.cinn_config'],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    noarchive=False,
    optimize=0,
)
pyz = PYZ(a.pure)

exe = EXE(
    pyz,
    a.scripts,
    [],
    exclude_binaries=True,
    name='test_ocr',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)
coll = COLLECT(
    exe,
    a.binaries,
    a.datas,
    strip=False,
    upx=True,
    upx_exclude=[],
    name='test_ocr',
)

Build

`1`	`pyinstaller ./test_ocr.spec`

What each section does

datas: include packages and non-Python assets required at runtime (e.g., ftfy, lxml, opencv-contrib-python).
binaries: include Paddle shared libraries under paddle/libs so the app runs without Paddle preinstalled.
hiddenimports: add modules PyInstaller may miss (e.g., cv2, paddle.cinn_config, ultra_infer).
UPX: reduces size. If startup issues occur, try upx=False.
One-folder vs one-file: this builds a folder via COLLECT; it’s more reliable for large frameworks.

PaddlePaddle Deployment on Ascend 310P: A Comprehensive Guide

Wed, 11 Jun 2025 00:00:00 +0000

Overview

This article provides a comprehensive guide for deploying PaddlePaddle models on the Ascend 310P platform. It covers multiple deployment approaches, their pros and cons, implementation steps, and solutions to common issues. The guide is particularly useful for ASR (Automatic Speech Recognition) and OCR (Optical Character Recognition) applications.

Deployment Approaches Overview

Approach	Description	Status	Recommendation
Approach 1	Using paddle-custom-npu pip package	❌ Dependency issues	⭐
Approach 2	Compiling PaddleCustomDevice	⚠️ Precision issues	⭐⭐
Approach 3	paddle2onnx + onnxruntime_cann	⚠️ Slow inference	⭐⭐⭐
Approach 4	paddle2onnx + OM model	⚠️ Dynamic shape issues	⭐⭐⭐
Approach 5	PaddleX High-Performance Inference	✅ Recommended	⭐⭐⭐⭐⭐

Approach 1: Pip Package Installation

Overview

This approach uses the official paddle-custom-npu pip package for quick deployment, but testing revealed missing runtime dependencies.

Deployment Steps

# Install basic dependencies
pip install psutil attrs decorator

# Install Python package dependencies
pip3 install py3Fdfs imageio pyheif whatimage shapely pyclipper minio \
    scikit-image imgaug lmdb pykafka gunicorn Pillow==9.5.0

# Install PaddlePaddle and NPU support
pip install paddlepaddle -i https://www.paddlepaddle.org.cn/packages/nightly/cpu/
pip install paddle-custom-npu -i https://www.paddlepaddle.org.cn/packages/nightly/npu/
pip install paddleocr

# Fix version compatibility issues
pip install protobuf==3.20.0

Configuration Modifications

# Modify PaddleSpeech executor configuration
vim /root/miniconda3/envs/asr_ocr/lib/python3.9/site-packages/paddlespeech/cli/executor.py +92

# Modify Paddle core module configuration
vim /root/miniconda3/envs/asr_ocr/lib/python3.9/site-packages/paddle/fluid/core.py +386

Issues Encountered

Missing runtime dependencies preventing normal inference
Internal errors persist even after adding related dependencies

Approach 2: Compilation Installation

Overview

Installing by compiling PaddleCustomDevice source code, but encountering precision support issues.

References

Environment Setup

# System dependencies installation
apt-get update -y && apt-get install -y \
    zlib1g zlib1g-dev libsqlite3-dev openssl libssl-dev libffi-dev \
    libbz2-dev libxslt1-dev unzip pciutils net-tools libblas-dev \
    gfortran libblas3 liblapack-dev liblapack3 libopenblas-dev git

# Python dependencies installation
pip install psutil attrs decorator pyyaml pathlib2 scipy requests \
    psutil absl-py sympy numpy==1.25.0 scipy

# CANN toolkit installation
./Ascend-cann-kernels-310p_8.0.RC2_linux.run --install
./Ascend-cann-nnal_8.0.RC2_linux-aarch64.run --install

# Operator package installation
wget -q https://paddle-ascend.bj.bcebos.com/code-share-master.zip --no-check-certificate
. /usr/local/Ascend/ascend-toolkit/set_env.sh
unzip code-share-master.zip
cd code-share-master/build && bash build_ops.sh
chmod +x aie_ops.run && ./aie_ops.run --extract=/usr/local/Ascend/

Environment Variables Configuration

# Log level configuration (0:debug 1:info 2:warning 3:error 4:null)
export ASCEND_GLOBAL_LOG_LEVEL=3

# HCCL configuration
export HCCL_CONNECT_TIMEOUT=7200
export HCCL_WHITELIST_DISABLE=1
export HCCL_SECURITY_MODE=1
export HCCL_BUFFSIZE=120

# PaddlePaddle NPU configuration
export FLAGS_npu_storage_format=0
export FLAGS_use_stride_kernel=0
export FLAGS_allocator_strategy=naive_best_fit
export PADDLE_XCCL_BACKEND=npu

Compilation and Installation

# Enter NPU backend directory
cd PaddleCustomDevice/backends/npu

# Install PaddlePaddle CPU version
pip install paddlepaddle -i https://www.paddlepaddle.org.cn/packages/nightly/cpu/

# Configure compilation options
export WITH_TESTING=OFF

# Execute compilation
bash tools/compile.sh

# Install compilation artifacts
pip install build/dist/paddle_custom_npu*.whl

Functionality Verification

# Check available hardware backends
python -c "import paddle; print(paddle.device.get_all_custom_device_type())"
# Expected output: ['npu']

# Check version information
python -c "import paddle_custom_device; paddle_custom_device.npu.version()"

# PaddlePaddle health check
python -c "import paddle; paddle.utils.run_check()"

Issues Encountered

Error: The soc version does not support bf16 / fp32 for calculations, please change the setting of cubeMathType or the Dtype of input tensor.

Approach 3: ONNX Runtime CANN Solution

Overview

Using paddle2onnx for model conversion, combined with onnxruntime_cann’s CANNExecutionProvider for inference.

Dependencies Installation

# Basic dependencies
pip install psutil attrs decorator pyyaml pathlib2 scipy requests \
    psutil absl-py sympy numpy==1.25.0 scipy packaging

# Image processing dependencies
pip install opencv-python Pillow==9.5.0

# Project dependencies
pip install flask py3Fdfs imageio pyheif whatimage shapely pyclipper \
    minio scikit-image imgaug lmdb pykafka gunicorn protobuf==3.20.0 \
    Pyinstaller nacos-python-sdk filetype

Model Conversion

# OCR recognition model conversion
paddle2onnx --model_dir ./ch_PP-OCRv4_rec_infer \
    --model_filename inference.pdmodel \
    --params_filename inference.pdiparams \
    --save_file ./ch_PP-OCRv4_rec_infer.onnx \
    --opset_version 11 \
    --enable_onnx_checker True

# OCR detection model conversion
paddle2onnx --model_dir ./ch_PP-OCRv4_det_infer \
    --model_filename inference.pdmodel \
    --params_filename inference.pdiparams \
    --save_file ./ch_PP-OCRv4_det_infer.onnx \
    --opset_version 11 \
    --enable_onnx_checker True

# Text direction classification model conversion
paddle2onnx --model_dir ./ch_ppocr_mobile_v2.0_cls_infer \
    --model_filename inference.pdmodel \
    --params_filename inference.pdiparams \
    --save_file ./ch_ppocr_mobile_v2.0_cls.onnx \
    --opset_version 11 \
    --enable_onnx_checker True

Performance Optimization

NPU inference performance optimization analysis:

Performance Issue Causes:

JIT Compilation Delay: NPU uses aclop JIT operator library, requiring operator compilation and caching on first run
Operator Fallback: Unsupported operators fall back to CPU, causing frequent memory copying

Optimization Solutions:

1
2
3

# Disable JIT compilation, use pre-compiled operators
export FLAGS_npu_jit_compile=0
export FLAGS_use_stride_kernel=0

Warm-up Preheating: Perform 5-10 warm-up runs before formal inference testing.

Operator Caching Mechanism: After execution, a kernel_meta directory is generated containing operator cache files to improve subsequent execution performance.

Issues Encountered

No significant inference speed improvement compared to CPU
Manual warm-up operations required
kernel_meta cache files may consume significant disk space

Approach 4: OM Model Deployment

Overview

Converting ONNX models to Ascend-specific OM model format for inference using ACL interfaces.

Model Conversion Process

# Step 1: Paddle model to ONNX
paddle2onnx --model_dir ./ch_PP-OCRv4_rec_infer \
    --model_filename inference.pdmodel \
    --params_filename inference.pdiparams \
    --save_file ./ch_PP-OCRv4_rec_infer.onnx \
    --opset_version 11 \
    --enable_onnx_checker True

# Step 2: ONNX model to OM
atc --model=./rec/ch_PP-OCRv4_rec_infer.onnx \
    --framework=5 \
    --input_format=NCHW \
    --output=./rec/ch_PP-OCRv4_rec_infer.om \
    --soc_version=Ascend310P3 \
    --input_shape="x:1,3,48,320"

Technical Challenges

Dynamic Shape Issues:

# Model information example
Input name  : x
Input shape : ['DynamicDimension.0', 3, 'DynamicDimension.1', 'DynamicDimension.2']
Output name : sigmoid_0.tmp_0  
Output shape: ['DynamicDimension.3', 1, 'DynamicDimension.4', 'DynamicDimension.5']

Memory Allocation Issues: get_output_size_by_index returns 0, causing acl.rt.malloc memory allocation failure.

Solutions

System Auto-allocation: Create empty aclDataBuffer, system automatically allocates memory
User Pre-allocation: Pre-allocate memory based on maximum possible output

Issues Encountered

Incomplete dynamic shape support
Complex memory management
Difficult debugging

Approach 5: PaddleX High-Performance Inference (Recommended)

Overview

Using PaddleX’s high-performance inference plugin, supporting fixed-shape OM model inference with excellent performance and stability.

Installation and Configuration

# Install PaddleX
git clone https://github.com/PaddlePaddle/PaddleX.git
cd PaddleX
pip install -e ".[base]"

# Install high-performance inference plugin
paddlex --install hpi-npu

Manual Compilation (Optional)

cd PaddleX/libs/ultra-infer/python
unset http_proxy https_proxy

# Configure compilation options
export ENABLE_OM_BACKEND=ON ENABLE_ORT_BACKEND=ON 
export ENABLE_PADDLE_BACKEND=OFF WITH_GPU=OFF DEVICE_TYPE=NPU
export NPU_HOST_LIB=/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux/lib64

# Compile and install
python setup.py build
python setup.py bdist_wheel
pip install dist/ultra_infer_npu*.whl

Model Conversion

paddlex --install paddle2onnx

# Convert to ONNX using PaddleX
paddlex --paddle2onnx \
    --opset_version 11 \
    --paddle_model_dir <PaddlePaddle_model_directory> \
    --onnx_model_dir <ONNX_model_directory>

# Convert to OM model
atc --model=inference.onnx \
    --framework=5 \
    --output=inference \
    --soc_version=Ascend310P3 \
    --input_shape "x:1,3,48,320"

# FP32 precision conversion
atc --model=inference.onnx \
    --framework=5 \
    --output=inference \
    --soc_version=Ascend310P3 \
    --input_shape "x:1,3,48,320" \
    --precision_mode_v2=origin

Inference Code Example

# -*- encoding=utf-8 -*-
"""
    @Author: Kang
    @Modified by: 
    @Datetime: 2025/07/15 13:57
    @Description: OCR test script
"""
import os
import time
import copy
from pathlib import Path
from typing import List, Dict, Any

import cv2
import numpy as np
from loguru import logger

from paddlex import create_model


class OCRProcessor:
    """OCR processor class"""
    
    def __init__(self, 
                 det_model_dir: str = "/opt/models/ocr/PP-OCRv4_server_det_infer_om_310P",
                 rec_model_dir: str = "/opt/models/ocr/PP-OCRv4_server_rec_infer_om_310P",
                 ori_model_dir: str = "/opt/models/ocr/PP-LCNet_x1_0_textline_ori_infer",
                 device: str = "npu:0",
                 output_dir: str = "/opt/output/"):
        
        self.output_dir = Path(output_dir)
        self.output_dir.mkdir(parents=True, exist_ok=True)
        
        # Configure parameters
        hpi_config = {
            "auto_config": False,
            "backend": "om",
        }
        
        # Initialize classification model
        logger.info("Loading classification model...")
        self.model_ori = create_model(
            model_name="PP-LCNet_x1_0_textline_ori",
            model_dir=ori_model_dir,
        )

        # Initialize detection model
        logger.info("Loading detection model...")
        self.model_det = create_model(
            model_name="PP-OCRv4_server_det",
            model_dir=det_model_dir,
            device=device,
            use_hpip=True,
            hpi_config=hpi_config,
            input_shape=[3, 640, 480]
        )
        
        # Initialize recognition model
        logger.info("Loading recognition model...")
        self.model_rec = create_model(
            model_name="PP-OCRv4_server_rec",
            model_dir=rec_model_dir,
            device=device,
            use_hpip=True,
            hpi_config=hpi_config,
            input_shape=[3, 48, 320]
        )
        logger.info("Model loaded successfully")
    
    def _crop_by_polys(self, img: np.ndarray, dt_polys: List[list]) -> List[dict]:
        """
        Call method to crop images based on detection boxes.

        Args:
            img (nd.ndarray): The input image.
            dt_polys (list[list]): List of detection polygons.

        Returns:
            list[dict]: A list of dictionaries containing cropped images and their sizes.

        Raises:
            NotImplementedError: If det_box_type is not 'quad' or 'poly'.
        """

        dt_boxes = np.array(dt_polys)
        output_list = []
        for bno in range(len(dt_boxes)):
            tmp_box = copy.deepcopy(dt_boxes[bno])
            img_crop = self.get_minarea_rect_crop(img, tmp_box)
            output_list.append(img_crop)

        return output_list

    def get_minarea_rect_crop(self, img: np.ndarray, points: np.ndarray) -> np.ndarray:
        """
        Get the minimum area rectangle crop from the given image and points.

        Args:
            img (np.ndarray): The input image.
            points (np.ndarray): A list of points defining the shape to be cropped.

        Returns:
            np.ndarray: The cropped image with the minimum area rectangle.
        """
        bounding_box = cv2.minAreaRect(np.array(points).astype(np.int32))
        points = sorted(list(cv2.boxPoints(bounding_box)), key=lambda x: x[0])

        index_a, index_b, index_c, index_d = 0, 1, 2, 3
        if points[1][1] > points[0][1]:
            index_a = 0
            index_d = 1
        else:
            index_a = 1
            index_d = 0
        if points[3][1] > points[2][1]:
            index_b = 2
            index_c = 3
        else:
            index_b = 3
            index_c = 2

        box = [points[index_a], points[index_b], points[index_c], points[index_d]]
        crop_img = self.get_rotate_crop_image(img, np.array(box))
        return crop_img

    def _rotate_image(self, image_array_list: List[np.ndarray], rotate_angle_list: List[int]):
        assert len(image_array_list) == len(
            rotate_angle_list
        ), f"Length of image ({len(image_array_list)}) must match length of angle ({len(rotate_angle_list)})"

        for angle in rotate_angle_list:
            assert angle in [0, 1], f"rotate_angle must be 0 or 1, now it's {angle}"

        rotated_images = []
        for image_array, rotate_indicator in zip(image_array_list, rotate_angle_list):
            # Convert 0/1 indicator to actual rotation angle
            rotate_angle = rotate_indicator * 180
            if rotate_angle < 0 or rotate_angle >= 360:
                raise ValueError("`angle` should be in range [0, 360)")

            if rotate_angle < 1e-7:
                rotated_images.append(image_array)
                continue

            # Should we align corners?
            h, w = image_array.shape[:2]
            center = (w / 2, h / 2)
            scale = 1.0
            mat = cv2.getRotationMatrix2D(center, rotate_angle, scale)
            cos = np.abs(mat[0, 0])
            sin = np.abs(mat[0, 1])
            new_w = int((h * sin) + (w * cos))
            new_h = int((h * cos) + (w * sin))
            mat[0, 2] += (new_w - w) / 2
            mat[1, 2] += (new_h - h) / 2
            dst_size = (new_w, new_h)

            rotated = cv2.warpAffine(
                image_array,
                mat,
                dst_size,
                flags=cv2.INTER_CUBIC,
            )
            rotated_images.append(rotated)
        logger.info(f"Number of rotated images: {len(rotated_images)}")
        return rotated_images

    def get_rotate_crop_image(self, img: np.ndarray, points: list) -> np.ndarray:
        """
        Crop and rotate the input image based on the given four points to form a perspective-transformed image.

        Args:
            img (np.ndarray): The input image array.
            points (list): A list of four 2D points defining the crop region in the image.

        Returns:
            np.ndarray: The transformed image array.
        """
        assert len(points) == 4, "shape of points must be 4*2"
        img_crop_width = int(
            max(
                np.linalg.norm(points[0] - points[1]),
                np.linalg.norm(points[2] - points[3]),
            )
        )
        img_crop_height = int(
            max(
                np.linalg.norm(points[0] - points[3]),
                np.linalg.norm(points[1] - points[2]),
            )
        )
        pts_std = np.float32(
            [
                [0, 0],
                [img_crop_width, 0],
                [img_crop_width, img_crop_height],
                [0, img_crop_height],
            ]
        )
        M = cv2.getPerspectiveTransform(points, pts_std)
        dst_img = cv2.warpPerspective(
            img,
            M,
            (img_crop_width, img_crop_height),
            borderMode=cv2.BORDER_REPLICATE,
            flags=cv2.INTER_CUBIC,
        )
        dst_img_height, dst_img_width = dst_img.shape[0:2]
        if dst_img_height * 1.0 / dst_img_width >= 1.5:
            dst_img = np.rot90(dst_img)
        return dst_img

    def process_image(self, image_path: str) -> List[Dict[str, Any]]:
        """
        Process single image for OCR recognition
        
        Args:
            image_path: Image path
            
        Returns:
            List of recognition results
        """
        if not os.path.exists(image_path):
            logger.error(f"Image file does not exist: {image_path}")
            return []
            
        try:
            # Read image
            image = cv2.imread(image_path)
            if image is None:
                logger.error(f"Failed to read image: {image_path}")
                return []
                
            logger.info(f"Start processing image: {image_path}")
            start_time = time.time()
            
            # Text detection
            logger.info(f"Start text detection")
            det_start = time.time()
            output_det = self.model_det.predict(image)
            det_time = time.time() - det_start
            logger.info(f"Text detection time: {det_time}s")
            
            results = []
            result_det = []
            for res in output_det:
                det_polys = res.get("dt_polys", [])
                det_scores = res.get("dt_scores", [])
                for idx, det_poly in enumerate(det_polys):
                    result_det.append({
                        "idx": idx,
                        "dt_polys": det_poly,
                        "dt_scores": det_scores[idx],
                    })
                logger.info(f"Detected {len(det_polys)} text regions")
            images_det = self._crop_by_polys(image, det_polys)
            for idx, img in enumerate(images_det):
                cv2.imwrite(f"{self.output_dir}/cropped_det_{idx}.png", img)
                    
            # Text direction classification
            logger.info(f"Start text direction classification")
            ori_start = time.time()
            output_ori = self.model_ori.predict(images_det)
            ori_time = time.time() - ori_start
            logger.info(f"Text direction classification time: {ori_time}s")
            angles = [
                int(ori_res["class_ids"][0])
                for ori_res in output_ori
            ]
            images_ori = self._rotate_image(images_det, angles)
            for idx, img in enumerate(images_ori):
                cv2.imwrite(f"{self.output_dir}/cropped_ori_{idx}.png", img)
            
            # Text recognition
            logger.info(f"Start text recognition")
            rec_start = time.time()
            for item in result_det:
                output_rec = self.model_rec.predict(images_ori[item["idx"]])
                for rec_res in output_rec:
                    rec_text = rec_res.get("rec_text", "")
                    rec_score = rec_res.get("rec_score", 0)
                    results.append({
                        "idx": item["idx"],
                        "dt_polys": item["dt_polys"].tolist(),
                        "dt_scores": item["dt_scores"],
                        "rec_res": rec_text,
                        "rec_score": rec_score,
                    })
            rec_time = time.time() - rec_start
            logger.info(f"Detection time-text recognition: {rec_time}s")

            total_time = time.time() - start_time
            logger.info(f"Total processing time: {total_time:.3f}s")
            
            return results
            
        except Exception as e:
            logger.exception(f"Failed to process image: {e}")
            return []
    
    def batch_process(self, image_paths: List[str]) -> Dict[str, List[Dict[str, Any]]]:
        """
        Batch process images
        
        Args:
            image_paths: List of image paths
            
        Returns:
            Batch processing results
        """
        batch_results = {}
        
        for image_path in image_paths:
            results = self.process_image(image_path)
            batch_results[image_path] = results
            
        return batch_results


def main():
    """Main function"""
    # Initialize OCR processor
    ocr_processor = OCRProcessor(
        output_dir="/opt/output/"
    )
    
    # Process single image
    image_path = "/opt/test/test.png"
    results = ocr_processor.process_image(image_path)
    
    # Output results
    print("\n=== OCR results ===")
    for result in results:
        print(result)


if __name__ == "__main__":
    main()

Supported Models

Model Type	Model Name	Input Shape	Chip Support
Text Detection	PP-OCRv4_mobile_det	(1,3,640,480)	910B/310P/310B
Text Recognition	PP-OCRv4_mobile_rec	(1,3,48,320)	910B/310P/310B
Image Classification	ResNet50	(1,3,224,224)	910B/310P/310B
Object Detection	RT-DETR-L	Multi-input	910B/310P/310B

Summary and Recommendations

Approach Comparison

Approach 5 (PaddleX): ✅ Highly Recommended - Official support, excellent performance, stable and reliable
Approach 3 (ONNX Runtime): ⚠️ Usable but average performance, suitable for quick validation
Approach 4 (OM Model): ⚠️ High technical difficulty, suitable for deep customization
Approaches 1 & 2: ❌ Not recommended due to multiple issues

Best Practices

Prioritize Approach 5: Use PaddleX high-performance inference plugin
Fixed Input Shapes: Avoid complexity introduced by dynamic shapes
Model Warm-up: Perform warm-up operations before inference
Performance Monitoring: Use Ascend CANN Profiling tools for performance optimization
Version Management: Keep CANN toolkit and PaddleX versions synchronized

Performance Optimization Recommendations

Use FP16 precision to improve inference speed
Configure batch size appropriately
Utilize operator caching mechanisms
Regularly clean up kernel_meta temporary files

Through this comprehensive guide, developers can select the appropriate deployment approach based on their specific requirements and reference corresponding troubleshooting methods to successfully deploy PaddlePaddle models on the Ascend 310P platform.