Kubernetes on Kang's Blog

Ascend NPU Deployment Guide for Kubernetes

Thu, 08 Jan 2026 00:00:00 +0000

Ascend NPU Deployment Guide for Kubernetes

Overview

This document describes the complete process of deploying the Ascend NPU containerized environment in a Kubernetes cluster, suitable for the following scenarios:

Container Runtime: Containerd
NPU Device: Ascend 310P
System Architecture: aarch64 (ARM64)
Kubernetes Version: 1.28+

The deployment process includes three main steps:

Environment preparation (node labels, users, directories)
Install Ascend Docker Runtime
Deploy Ascend Device Plugin

Preparation

Create Node Labels

Add appropriate labels to Kubernetes nodes for subsequent Pod scheduling and resource management.

# Label master node
kubectl label nodes ecs-b0tf90001 masterselector=dls-master-node

# Label NPU compute nodes
kubectl label nodes ecs-exyqec0002 node-role.kubernetes.io/worker=worker
kubectl label nodes ecs-exyqec0002 workerselector=dls-worker-node
kubectl label nodes ecs-exyqec0002 host-arch=huawei-arm
kubectl label nodes ecs-exyqec0002 accelerator=huawei-Ascend310P

Note: Please modify ecs-b0tf90001 and ecs-exyqec0002 according to your actual node names.

Create System User

Create dedicated users and groups on Ascend compute nodes (e.g., ecs-exyqec0002).

# Create hwMindX user (UID 9000)
useradd -d /home/hwMindX -u 9000 -m -s /sbin/nologin hwMindX

# Create HwHiAiUser group
groupadd HwHiAiUser

# Add hwMindX user to HwHiAiUser group
usermod -a -G HwHiAiUser hwMindX

Important: UID 9000 and group HwHiAiUser are the default configurations for the Ascend software stack. Do not modify them arbitrarily.

Create Log Directory

Create the Device Plugin log directory on Ascend compute nodes.

1
2
3

# Create log directory and set permissions
mkdir -m 750 /var/log/mindx-dl/devicePlugin
chown root:root /var/log/mindx-dl/devicePlugin

Install Ascend Docker Runtime

Ascend Docker Runtime is a core component for using Ascend NPU in containerized environments and must be installed on all Ascend compute nodes.

Download Installation Package

Visit the official Git repository to download the corresponding version of the installation package.

Example: Ascend-docker-runtime_7.2.RC1.SPC2_linux-aarch64.run

Installation Steps

Step 1: Enter Installation Package Directory

`1`	`cd <path to run package>`

Step 2: Verify Package Integrity

`1`	`./Ascend-docker-runtime_{version}_linux-{arch}.run --check`

Expected output:

1
2

[WARNING]: --check is meaningless for Ascend-docker-runtime and will be discarded in the future
Verifying archive integrity... All good.

Step 3: Add Executable Permission

`1`	`chmod u+x Ascend-docker-runtime_{version}_linux-{arch}.run`

Step 4: Execute Installation

Method 1: Install to Default Path (Recommended)

`1`	`./Ascend-docker-runtime_{version}_linux-{arch}.run --install`

Method 2: Install to Custom Path

`1`	`./Ascend-docker-runtime_{version}_linux-{arch}.run --install --install-path=<path>`

Successful installation output example:

Uncompressing ascend-docker-runtime  100%
[INFO]: installing ascend docker runtime
...
[INFO] Ascend Docker Runtime install success

Default Installation Path: /usr/local/Ascend/Ascend-Docker-Runtime/

Configure Containerd

Modify the Containerd configuration file /etc/containerd/config.toml according to the cgroup version used by your system.

Configuration Method 1: Cgroup v1

Two key configuration items need to be modified:

runtime_type = "io.containerd.runtime.v1.linux"
runtime = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"

Complete configuration example:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runtime.v1.linux"
    runtime_engine = ""
    runtime_root = ""
    privileged_without_host_devices = false
    base_runtime_spec = ""
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]

[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/opt/cni/bin"
  conf_dir = "/etc/cni/net.d"
  max_conf_num = 1
  conf_template = ""

[plugins."io.containerd.grpc.v1.cri".registry]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["https://registry-1.docker.io"]

[plugins."io.containerd.grpc.v1.cri".image_decryption]
  key_model = ""

# ... other configurations ...

[plugins."io.containerd.monitor.v1.cgroups"]
  no_prometheus = false

[plugins."io.containerd.runtime.v1.linux"]
  shim = "containerd-shim"
  runtime = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"
  runtime_root = ""
  no_shim = false
  shim_debug = false

[plugins."io.containerd.runtime.v2.task"]
  platforms = ["linux/amd64"]

Configuration Method 2: Cgroup v2

The following key configuration item needs to be modified:

BinaryName = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"

Complete configuration example:

[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
  [plugins."io.containerd.grpc.v2.cri".containerd.runtimes.runc]
    base_runtime_spec = ""
    cni_conf_dir = ""
    cni_max_conf_num = 0
    container_annotations = []
    pod_annotations = []
    privileged_without_host_devices = false
    runtime_engine = ""
    runtime_path = ""
    runtime_root = ""
    runtime_type = "io.containerd.runc.v2"
    
    [plugins."io.containerd.grpc.v2.cri".containerd.runtimes.runc.options]
      BinaryName = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"
      CriuImagePath = ""
      CriuPath = ""
      CriuWorkPath = ""
      IoGid = 0
      IoUid = 0
      NoNewKeyring = false
      NoPivotRoot = false
      Root = ""
      ShimCgroup = ""
      SystemdCgroup = true

Tip: To check the cgroup version, execute stat -fc %T /sys/fs/cgroup/. Output cgroup2fs indicates v2, while tmpfs indicates v1.

Restart Services

1
2

systemctl daemon-reload
systemctl restart containerd kubelet

Verify Installation

Execute the following command on the Kubernetes master node to confirm that the Ascend compute nodes are in normal status:

`1`	`kubectl get nodes`

Expected output example:

1
2
3

NAME              STATUS   ROLES           AGE   VERSION
k8s-master        Ready    master,worker   3d    v1.28.12
k8s-worker        Ready    worker          3d    v1.28.12

All nodes should show Ready status.

Deploy Ascend Device Plugin

Ascend Device Plugin is responsible for managing and allocating NPU resources in Kubernetes.

Prepare Images

1. Pull Images

Execute the following commands on Ascend compute nodes to pull the required images:

# Pull all required images
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/resilience-controller:v7.1.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-operator:v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/npu-exporter:v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-k8sdeviceplugin:v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-controller-manager:v1.7.0-v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-scheduler:v1.7.0-v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/noded:v7.2.RC1
docker pull --platform=arm64 swr.cn-south-1.myhuaweicloud.com/ascendhub/clusterd:v7.2.RC1

2. Export Images

docker save -o ascend.tar \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/resilience-controller:v7.1.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-operator:v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/npu-exporter:v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-k8sdeviceplugin:v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-controller-manager:v1.7.0-v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-scheduler:v1.7.0-v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/noded:v7.2.RC1 \
  swr.cn-south-1.myhuaweicloud.com/ascendhub/clusterd:v7.2.RC1

3. Import to Containerd

`1`	`ctr -n k8s.io images import ascend.tar`

Download Deployment Configuration Files

Visit the official Git repository to download the Ascend Device Plugin installation package.

Example: Ascend-mindxdl-device-plugin_7.2.RC1.SPC2_linux-aarch64.zip

After extracting, copy the corresponding YAML files to the Kubernetes management node.

Choose the Appropriate YAML File

Select the corresponding YAML file based on the actual device type and whether to use the Volcano scheduler:

YAML Filename	Use Case
`device-plugin-310-v{version}.yaml`	Atlas 300I inference card, without Volcano
`device-plugin-310-volcano-v{version}.yaml`	Atlas 300I inference card, with Volcano
`device-plugin-310P-1usoc-v{version}.yaml`	Atlas 200I SoC A1 core board, without Volcano
`device-plugin-310P-1usoc-volcano-v{version}.yaml`	Atlas 200I SoC A1 core board, with Volcano
`device-plugin-310P-v{version}.yaml`	Atlas inference series products (e.g., 310P), without Volcano
`device-plugin-310P-volcano-v{version}.yaml`	Atlas inference series products, with Volcano
`device-plugin-910-v{version}.yaml`	Atlas training series products/A2/A3/800I A2, without Volcano
`device-plugin-volcano-v{version}.yaml`	Atlas training series products/A2/A3/800I A2, with Volcano

Note:

For Ascend 310P devices, typically choose device-plugin-310P-v{version}.yaml

Do not modify the DaemonSet.metadata.name field in the YAML file to avoid automatic identification issues

Deploy Device Plugin

1
2

# Please modify the image name to: swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-k8sdeviceplugin:v7.2.RC1
kubectl apply -f device-plugin-310P-v7.2.RC1.SPC2.yaml

Expected output:

serviceaccount/ascend-device-plugin-sa created
clusterrole.rbac.authorization.k8s.io/pods-node-ascend-device-plugin-role created
clusterrolebinding.rbac.authorization.k8s.io/pods-node-ascend-device-plugin-rolebinding created
daemonset.apps/ascend-device-plugin-daemonset created

Verify Deployment

Check if the Device Plugin started successfully:

`1`	`kubectl get pod -n kube-system \| grep ascend`

Expected output (status should be Running):

1
2

NAME                                   READY   STATUS    RESTARTS   AGE
ascend-device-plugin-daemonset-d5ctz   1/1     Running   0          11s

Check node NPU resources:

`1`	`kubectl describe node <node-name> \| grep -A 5 "Capacity:"`

You should see the huawei.com/Ascend310P resource.

Using NPU Compute Cards

Pod Configuration Example

Request NPU resources through the resources field in the Pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: npu-test-pod
spec:
  containers:
    - name: alg-container
      image: ubuntu:22.04
      resources:
        limits:
          memory: 24Gi
          huawei.com/Ascend310P: 1  # Request 1 NPU card
        requests:
          memory: 2Gi
          huawei.com/Ascend310P: 1  # Request 1 NPU card
      command: ["/bin/bash", "-c", "sleep infinity"]

Note:

The value of huawei.com/Ascend310P indicates the number of NPU cards requested

Values in limits and requests should be consistent

Adjust the resource name according to the actual NPU model (e.g., 310, 910, etc.)

Kubernetes Package Management: The Ultimate Helm Guide

Sat, 18 Jan 2025 00:00:00 +0000

Introduction

Helm is a powerful tool for managing Kubernetes applications. Through Helm Charts, you can define, install, and upgrade even the most complex Kubernetes applications.

Helm Charts are easy to create, version control, share, and publish. This means you can use Helm more efficiently, avoiding the tedious process of manually copying and pasting configuration files.

Helm has graduated from CNCF (Cloud Native Computing Foundation) and is continuously maintained by the Helm community, ensuring its stability and ongoing development.

Managing Complexity

Helm Charts can describe even the most complex applications, providing reproducible installation workflows and serving as the single source of truth for application configuration to ensure consistency and reliability.
Easy Updates

With Helm, you can effortlessly manage application updates through in-place upgrades and custom hooks, minimizing disruptions caused by updates.
Simple Sharing

Helm Charts are easy to version control, share, and host on public or private repositories, facilitating team collaboration and application distribution.
Rollbacks

If issues arise during updates, the helm rollback command allows you to easily revert to a previous release version, ensuring stable application operation.

Helm’s Workflow

The workflow of Helm can be understood as follows:

Install Charts: Helm installs Charts into a Kubernetes cluster, creating a new Release each time.
Manage Releases: Each Release is independently managed, allowing the same Chart to be deployed multiple times to meet different requirements.
Find Charts: Charts can be found and obtained through Helm’s Chart Repository, allowing for new Charts to be deployed.

Overall Diagram Representation:

+---------------------+          +---------------------+
|    Chart Repository | <------> |        Helm         |
| (Store all Charts)    |          | (Install and Manage Charts)   |
+---------------------+          +---------+-----------+
                                         |
                                         v
                               +--------------------+
                               | Kubernetes Cluster |
                               | ------------------ |
                               | Release A          |
                               | Release B          |
                               | ...                |
                               +--------------------+

Three Core Concepts

Chart

Chart is a Helm package. It contains all the resource definitions required to run an application, tool, or service in a Kubernetes cluster. It can be compared to the Kubernetes version of Homebrew recipes, dpkg, or RPM files in Apt or Yum.

Diagram Representation:

+-----------------+
| Chart           |
| --------------- |
| Deployment.yaml |
| Service.yaml    |
| ConfigMap.yaml  |
| ...             |
+-----------------+

Helm will deploy the Chart in the following order:

Namespace —> NetworkPolicy —> ResourceQuota —> LimitRange —> PodSecurityPolicy —> PodDisruptionBudget —> ServiceAccount —> Secret —> SecretList —> ConfigMap —> StorageClass —> PersistentVolume —> PersistentVolumeClaim —> CustomResourceDefinition —> ClusterRole —> ClusterRoleList —> ClusterRoleBinding —> ClusterRoleBindingList —> Role —> RoleList —> RoleBinding —> RoleBindingList —> Service —> DaemonSet —> Pod —> ReplicationController —> ReplicaSet —> Deployment —> HorizontalPodAutoscaler —> StatefulSet —> Job —> CronJob —> Ingress —> APIService

Repository

Repository is a collection of Charts. It can be compared to the Apt or Yum repository.

Diagram Representation:

+---------------------+
| Chart Repository    |
| ------------------- |
| Chart A     Chart B |
| Chart C     Chart D |
| ...                 |
+---------------------+

Release

Release is a running instance of a Chart. A Chart can be installed multiple times in a Kubernetes cluster, each with its own Release. For example, if you have a MySQL Chart, you can install it twice in the same cluster, each with its own Release.

Diagram Representation:

+------------------+       +------------------+
| Release A 	   |       | Release B        |
|------------------|       |------------------|
| MySQL Chart      |       | MySQL Chart      |
| Release Name: A  |       | Release Name: B  |
| ...              |       | ...              |
+------------------+       +------------------+

Installation

Each release of Helm provides binary releases for a variety of OSes. These binary versions can be manually downloaded and installed.

Download your desired version
Unpack it (tar -zxvf helm-v3.0.0-linux-amd64.tar.gz)
Find the helm binary in the unpacked directory, and move it to its desired destination (mv linux-amd64/helm /usr/local/bin/helm)

Create Project Templates

Create a Helm project template using the command:

`1`	`helm create mychart`

This generates the following files and directory structure:

.
├── charts                          # Directory for dependency Charts
├── Chart.yaml                      # Metadata describing the Chart
├── templates                       # Directory for template files
│   ├── deployment.yaml             # Deployment template
│   ├── _helpers.tpl                # Helper templates
│   ├── hpa.yaml                    # Horizontal Pod Autoscaler template
│   ├── ingress.yaml                # Ingress template
│   ├── NOTES.txt                   # Installation notes
│   ├── serviceaccount.yaml         # Service Account template
│   ├── service.yaml                # Service template
│   └── tests                       # Test templates
│       └── test-connection.yaml    # Connection test template
└── values.yaml                     # Default values overriding template parameters

Summary

Helm simplifies the deployment and management of applications, tools, and services in Kubernetes clusters through three core concepts:

Charts: Define applications
Repositories: Enable sharing and distribution of Charts
Releases: Manage running instances of these applications in clusters

The charts provide application definitions, repositories facilitate sharing, and releases handle the lifecycle of deployed instances.

Guide to Configuring TCP Service Forwarding with Ingress-Nginx in Kubernetes

Fri, 20 Dec 2024 00:00:00 +0000

This guide explains how to configure an Ingress-Nginx controller in a Kubernetes cluster to support TCP service forwarding, using MySQL service as an example.

Prerequisites

Kubernetes cluster installed
Ingress-Nginx controller deployed
Existing TCP service to expose (MySQL in this example)

Configuration Steps

Configure TCP Service Mapping

Create a tcp-services.yaml file to define TCP port mappings:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  # Syntax: <external_port>: "<namespace>/<service_name>:<target_port>"
  3306: "default/mysql-primary:3306"

Apply configuration:

`1`	`kubectl create -f tcp-services.yaml`

If the ConfigMap already exists, edit directly:

1
2

kubectl edit configmap tcp-services -n ingress-nginx
# Add new mapping under data: <external_port>: "<namespace>/<service_name>:<port>"

Update Ingress Controller Configuration

Edit the Ingress-Nginx Controller Deployment:

`1`	`kubectl edit deployment ingress-nginx-controller -n ingress-nginx`

Add the following configuration to the controller’s args section:

1
2
3

args:
  # ... existing parameters ...
  - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

Configuring Ingress Controller Service

Edit the Ingress-Nginx Controller Service:

`1`	`kubectl edit service ingress-nginx-controller -n ingress-nginx`

Add TCP service port configuration in the ports section:

ports:
  # ... Other existing port configurations ...
  - name: mysql-primary
    port: 3306
    protocol: TCP
    targetPort: 3306

Verification

After completing configurations, verify with following methods:

Check ConfigMap creation:

`1`	`kubectl get configmap tcp-services -n ingress-nginx -o yaml`

Confirm Ingress Controller status:

`1`	`kubectl get pods -n ingress-nginx`

Verify port exposure:

`1`	`kubectl get svc ingress-nginx-controller -n ingress-nginx`

Notes

Ensure target services (e.g., MySQL) are properly running in the cluster
Check for port conflicts
Additional security group/firewall rule configurations may be required for cloud providers

Kubernetes Ingress-NGINX Service Certificate Configuration

Mon, 16 Dec 2024 00:00:00 +0000

Using Ingress | Kubernetes as the application traffic proxy for K8s, and configuring certificates

Service Deployment

Since the K8S cluster used is not provided by a cloud provider, deploying metallb is required when using LoadBalancer-type services for ingress-nginx. MetalLB is a load balancer implementation for bare-metal Kubernetes clusters, using standard routing protocols.

Deploy MetalLB

# Preview configuration changes (returns non-zero if changes detected)  
kubectl get configmap kube-proxy -n kube-system -o yaml | \  
sed -e "s/strictARP: false/strictARP: true/" | \  
kubectl diff -f - -n kube-system  

# Apply ARP configuration changes (returns non-zero only on errors)  
kubectl get configmap kube-proxy -n kube-system -o yaml | \  
sed -e "s/strictARP: false/strictARP: true/" | \  
kubectl apply -f - -n kube-system

1
2

# Install MetalLB components  
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml

Create ip-pool.yaml to define the IP address pool:

apiVersion: metallb.io/v1beta1  
kind: IPAddressPool  
metadata:  
  name: default  
  namespace: metallb-system  
spec:  
  addresses:  
  - 172.16.0.90/32  # Manual IP assignment required for bare-metal clusters  
  autoAssign: true  
---  
apiVersion: metallb.io/v1beta1  
kind: L2Advertisement  
metadata:  
  name: default  
  namespace: metallb-system  
spec:  
  ipAddressPools:  
  - default

Deploy ingress-nginx

`1`	`kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.2/deploy/static/provider/cloud/deploy.yaml`

Configuring Certificates

Here we introduce 3 certificate configuration methods: manually generating self-signed certificates, manually obtaining trusted certificates, and using cert-manager to acquire certificates.

Manual Self-Signed Certificate

You can use OpenSSL or other tools to apply for certificates. The tls.crt certificate file and tls.key private key file, the base64 encoded content of the certificate and private key, can customize the relevant information of the certificate: country, province, city, company, department, domain name, validity period, etc.

# Generate private key
openssl genrsa -out tls.key 2048 # Generate private key file tls.key

# Generate certificate request
openssl req -new -key tls.key -out tls.csr -subj "/C=CN/ST=<state>/L=<city>/O=<company>/OU=<department>/CN=<domain>" # Generate certificate request file tls.csr

# Generate certificate
openssl x509 -req -in tls.csr -signkey tls.key -out tls.crt -days 3650 # Generate certificate file tls.crt

# View certificate information
openssl x509 -in tls.crt -text -noout

test-ingress.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: test-ingress


---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service1
  namespace: test-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service1
  template:
    metadata:
      labels:
        app: service1
    spec:
      containers:
      - name: service1
        image: nginx:alpine


---
apiVersion: v1
kind: Service
metadata:
  name: service1
  namespace: test-ingress
spec:
  selector:
    app: service1
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80


# letsencrypt certificate (manually created)
---
apiVersion: v1
kind: Secret
metadata:
  name: example-com
  namespace: test-ingress
data:
  tls.crt: <base64 encoded cert>
  tls.key: <base64 encoded key>
type: kubernetes.io/tls


---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test-ingress
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - test.example.com
      secretName: example-com
  rules:
    - host: test.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service1
                port:
                  number: 80

`1`	`kubectl apply -f test-ingress.yaml`

Manual Trusted Certificate Application

To apply for a certificate, you can use Let’s Encrypt or other Certificate Authorities (CAs).
The TLS certificate file (tls.crt) and private key file (tls.key) must be Base64-encoded. The default validity period is 90 days.

test-ingress.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: test-ingress

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service1
  namespace: test-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service1
  template:
    metadata:
      labels:
        app: service1
    spec:
      containers:
      - name: service1
        image: nginx:alpine

---
apiVersion: v1
kind: Service
metadata:
  name: service1
  namespace: test-ingress
spec:
  selector:
    app: service1
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80

# letsencrypt certificate (manually created)
---
apiVersion: v1
kind: Secret
metadata:
  name: example-com
  namespace: test-ingress
data:
  tls.crt: <base64 encoded cert>
  tls.key: <base64 encoded key>
type: kubernetes.io/tls

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test-ingress
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - test.example.com
      secretName: example-com
  rules:
    - host: test.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service1
                port:
                  number: 80

`1`	`kubectl apply -f test-ingress.yaml`

Using cert-manager to Obtain Certificates

cert-manager is a Kubernetes certificate management controller that utilizes CustomResourceDefinitions (CRDs), offering features such as certificate application, issuance, renewal, and deletion. Below are three methods for obtaining certificates using cert-manager: SelfSigned Issuer-type certificates, ACME Issuer-type certificates, and CA Issuer-type certificates.

Installing cert-manager

`1`	`kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.3/cert-manager.yaml`

By default, cert-manager will be installed in the cert-manager namespace. Although it can be deployed in a different namespace, this requires modifications to the deployment manifests.

After installation, verify the deployment by checking the pods in the cert-manager namespace:

$ kubectl get pods --namespace cert-manager

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-cainjector-5fd6444f95-kmbmd   1/1     Running   0          60m
cert-manager-d894bbbd4-lrwp5               1/1     Running   0          60m
cert-manager-webhook-869674f96f-ljffr      1/1     Running   0          60m

Configuring SelfSigned Issuer Type Certificates

test-ingress.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: test-ingress


---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service1
  namespace: test-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service1
  template:
    metadata:
      labels:
        app: service1
    spec:
      containers:
      - name: service1
        image: nginx:alpine


---
apiVersion: v1
kind: Service
metadata:
  name: service1
  namespace: test-ingress
spec:
  selector:
    app: service1
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80


# SelfSigned Issuer: 100 years, automatically created
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
  namespace: test-ingress
spec:
  selfSigned: {}

# SelfSigned Issuer: 100 years, automatically created
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com
  namespace: test-ingress
spec:
  secretName: example-com
  duration: 876000h  # 100 years
  renewBefore: 720h  # The certificate will be renewed 30 days before expiration
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
  commonName: test.example.com
  subject:
    organizations:
      - '*** Technology Co., Ltd.'
    organizationalUnits:
      - '*** Technology Co., Ltd. Operations Department'
  isCA: true
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048


---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test-ingress
  annotations:
    cert-manager.io/cluster-issuer: selfsigned-issuer
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - test.example.com
      secretName: example-com
  rules:
    - host: test.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service1
                port:
                  number: 80

`1`	`kubectl apply -f test-ingress.yaml`

ACME (Automated Certificate Management Environment) is a protocol used for automated certificate issuance and renewal. The ACME protocol is standardized by the IETF, with the most common current implementation being Let’s Encrypt. A key feature of the ACME protocol is that it allows Certificate Authorities (CAs) to verify the identity of certificate requesters without requiring human intervention. Another critical feature of the ACME protocol is its ability to enable automatic certificate renewal by CAs without manual involvement. The Issuer type represents an individual account registered with an ACME certificate authority server. When creating a new ACME Issuer, the Certificate Manager will generate a private key used for identification on the ACME server. By default, certificates issued by public ACME servers are typically trusted by client computers. This means that, for example, websites secured by ACME certificates issued for specific URLs will be automatically trusted by most web browsers. ACME certificates are generally free.

Configuring ACME Issuer with HTTP01 Certificate Type

The HTTP01 challenge is completed by exposing a computed secret key at a publicly accessible HTTP URL endpoint. This URL will use the domain name for which the certificate is being requested. Once the ACME server can retrieve this secret key from the URL via the internet, it verifies your ownership of the domain.

When creating an HTTP01 challenge, cert-manager will automatically configure your cluster ingress to route traffic accessing this URL to a small web server that presents the secret key. In simpler terms, cert-manager automatically creates an ingress route to a lightweight web server responsible for displaying the verification key, thereby proving domain ownership.

Note:

Fake certificate (1-year validity, auto-created): kubernetes ingress controller fake cert
Example file: test-ingress.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: test-ingress

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service1
  namespace: test-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service1
  template:
    metadata:
      labels:
        app: service1
    spec:
      containers:
      - name: service1
        image: nginx:alpine

---
apiVersion: v1
kind: Service
metadata:
  name: service1
  namespace: test-ingress
spec:
  selector:
    app: service1
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80

# ACME certificate, automatically created
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: sencrypt-prod
  namespace: test-ingress
spec:
  acme:
    email: CoderKang@hotmail.com
    privateKeySecretRef:
      name: sencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
      - http01:
          ingress:
            class: nginx

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test-ingress
  annotations:
    cert-manager.io/cluster-issuer: sencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - test.example.com
      secretName: example-com
  rules:
    - host: test.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service1
                port:
                  number: 80

`1`	`kubectl apply -f test-ingress.yaml`

Configuring ACME Issuer dns01-type Certificates

The DNS01 challenge is completed by providing a cryptographic key that exists in a DNS TXT record. Once this TXT record propagates across the internet, the ACME server can retrieve the key via DNS queries and verify that the client requesting the certificate is the domain owner.
If proper permissions are granted, cert-manager will automatically add this TXT record to your specified DNS provider.

However, since Tencent Cloud DNS is used here, and cert-manager does not natively support Tencent Cloud DNS, a custom resolver (cert-manager webhook) is required.

DNS01 - cert-manager Documentation

`1`	`kubectl apply -f https://raw.githubusercontent.com/imroc/cert-manager-webhook-dnspod/master/bundle.yaml`

test-ingress.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: test-ingress


---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service1
  namespace: test-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service1
  template:
    metadata:
      labels:
        app: service1
    spec:
      containers:
      - name: service1
        image: nginx:alpine


---
apiVersion: v1
kind: Service
metadata:
  name: service1
  namespace: test-ingress
spec:
  selector:
    app: service1
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80


# letsencrypt automatically created, valid for 90 days
# Configure the SecretId and SecretKey of the Tencent Cloud DNS provider
# https://console.dnspod.cn/account/token/apikey
---
apiVersion: v1
stringData:
  secret-key: <tencent cloud secret key>  # Tencent Cloud SecretKey
kind: Secret
metadata:
  name: dnspod-secret
  namespace: cert-manager
type: Opaque

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: sencrypt-prod
  namespace: test-ingress
spec:
  acme:
    email: CoderKang@hotmail.com
    preferredChain: ""
    privateKeySecretRef:
      name: dnspod-letsencrypt
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
      - dns01:
          webhook:
            config:
              secretId: <tencent cloud secret id>  # Tencent Cloud SecretId
              secretKeyRef:
                key: secret-key
                name: dnspod-secret
              ttl: 600
            groupName: acme.imroc.cc
            solverName: dnspod

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com
  namespace: test-ingress
spec:
  dnsNames:
    - test.example.com
  issuerRef:
    kind: ClusterIssuer
    name: sencrypt-prod
  secretName: example-com

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test-ingress
  annotations:
    cert-manager.io/cluster-issuer: sencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - test.example.com
      secretName: example-com
  rules:
    - host: test.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service1
                port:
                  number: 80

`1`	`kubectl apply -f test-ingress.yaml`

Deploying JumpServer on Kubernetes with DingTalk Alert Integration

Tue, 19 Nov 2024 00:00:00 +0000

Ingress Configuration

Requires setting up ingress (see 13-K8s ingress-nginx Service Certificate) and enabling TCP forwarding (see 23-K8s ingress TCP Forwarding).

JumpServer Deployment

“Installation”

`1`	`helm install jms-k8s ./jumpserver-v4.6.0 -n jumpserver --create-namespace -f values.yaml`

“Upgrade”

`1`	`helm upgrade jms-k8s ./jumpserver-v4.6.0 -n jumpserver --create-namespace -f values.yaml`

“Uninstallation”

`1`	`helm -n jumpserver delete jms-k8s`

Core Component Image

Modified core component to support DingTalk alerts.

docker pull docker.1ms.run/jumpserver/core:v4.6.0-ce

cd core
docker build -t jumpserver/core:v4.6.1-ce .

“Dockerfile”

1
2
3

    FROM docker.1ms.run/jumpserver/core:v4.6.0-ce

    COPY ./notifications.py /opt/jumpserver/apps/notifications/notifications.py

“notifications.py”

import textwrap
import time
import traceback
from itertools import chain

from celery import shared_task
from django.utils.translation import gettext_lazy as _
from html2text import HTML2Text

from common.utils import lazyproperty
from common.utils.timezone import local_now
from notifications.backends import BACKEND
from settings.utils import get_login_title
from terminal.const import RiskLevelChoices
from users.models import User
from .models import SystemMsgSubscription, UserMsgSubscription

__all__ = ('SystemMessage', 'UserMessage', 'system_msgs', 'Message')

system_msgs = []
user_msgs = []


class MessageType(type):
    def __new__(cls, name, bases, attrs: dict):
        clz = type.__new__(cls, name, bases, attrs)

        if 'message_type_label' in attrs \
                and 'category' in attrs \
                and 'category_label' in attrs:
            message_type = clz.get_message_type()

            msg = {
                'message_type': message_type,
                'message_type_label': attrs['message_type_label'],
                'category': attrs['category'],
                'category_label': attrs['category_label'],
            }
            if issubclass(clz, SystemMessage):
                system_msgs.append(msg)
            elif issubclass(clz, UserMessage):
                user_msgs.append(msg)

        return clz


@shared_task(
    verbose_name=_('Publish the station message'),
    description=_(
        """This task needs to be executed for sending internal messages for system alerts, 
        work orders, and other notifications"""
    )
)
def publish_task(receive_user_ids, backends_msg_mapper):
    Message.send_msg(receive_user_ids, backends_msg_mapper)


def send_dingtalk_task(message):
    # Send to custom webhook
    import hmac, hashlib, base64, urllib.parse
    timestamp = str(round(time.time() * 1000))
    # JumpServer credentials
    access_token = '<access token>'
    secret = '<secret>'

    secret_enc = secret.encode('utf-8')
    string_to_sign = f'{timestamp}\n{secret}'
    string_to_sign_enc = string_to_sign.encode('utf-8')
    hmac_code = hmac.new(secret_enc, string_to_sign_enc, digestmod=hashlib.sha256).digest()
    sign = urllib.parse.quote_plus(base64.b64encode(hmac_code))
    form_data = {
        "msgtype": "markdown",
        "markdown": {
            "title": "JumpServer Monitoring Alert",
            "text": message
        },
        "at": {
            "atMobiles": [
                "<mobile>"
            ],
            "isAtAll": False
        }
    }
    import requests
    try:
        res = requests.post(
            url=f'https://oapi.dingtalk.com/robot/send?access_token={access_token}&timestamp={timestamp}&sign={sign}',
            json=form_data)
    except Exception as e:
        with open('/opt/error.log', 'a+') as f:
            f.write(str(e))


class Message(metaclass=MessageType):
    """
    What's encapsulated here?
        Templates for different messages with a unified sending interface
        - publish: Implementation relates to message subscription table structure
        - send_msg
    """
    message_type_label: str
    category: str
    category_label: str
    text_msg_ignore_links = True
    command = None

    @classmethod
    def get_message_type(cls):
        return cls.__name__

    def publish_async(self):
        self.publish(is_async=True)

    @classmethod
    def gen_test_msg(cls):
        raise NotImplementedError

    def publish(self, is_async=False):
        raise NotImplementedError

    def get_backend_msg_mapper(self, backends):
        backends = set(backends)
        backends.add(BACKEND.SITE_MSG)  # Site message is mandatory
        backends_msg_mapper = {}
        for backend in backends:
            backend = BACKEND(backend)
            if not backend.is_enable:
                continue
            get_msg_method = getattr(self, f'get_{backend}_msg', self.get_common_msg)
            msg = get_msg_method()
            backends_msg_mapper[backend] = msg
        return backends_msg_mapper

    @staticmethod
    def send_msg(receive_user_ids, backends_msg_mapper):
        for backend, msg in backends_msg_mapper.items():
            try:
                backend = BACKEND(backend)
                client = backend.client()
                users = User.objects.filter(id__in=receive_user_ids).all()
                client.send_msg(users, **msg)
            except NotImplementedError:
                continue
            except:
                traceback.print_exc()

    @classmethod
    def send_test_msg(cls, ding=True, wecom=False):
        msg = cls.gen_test_msg()
        if not msg:
            return

        from users.models import User
        users = User.objects.filter(username='admin')
        backends = []
        if ding:
            backends.append(BACKEND.DINGTALK)
        if wecom:
            backends.append(BACKEND.WECOM)
        msg.send_msg(users, backends)

    @staticmethod
    def get_common_msg() -> dict:
        return {'subject': '', 'message': ''}

    def get_html_msg(self) -> dict:
        return self.get_common_msg()

    @staticmethod
    def html_to_markdown(html_msg):
        h = HTML2Text()
        h.body_width = 0
        content = html_msg['message']
        html_msg['message'] = h.handle(content)
        return html_msg

    def get_markdown_msg(self) -> dict:
        return self.html_to_markdown(self.get_html_msg())

    def get_text_msg(self) -> dict:
        h = HTML2Text()
        h.body_width = 90
        msg = self.get_html_msg()
        content = msg['message']
        h.ignore_links = self.text_msg_ignore_links
        msg['message'] = h.handle(content)
        return msg

    @lazyproperty
    def common_msg(self) -> dict:
        return self.get_common_msg()

    @lazyproperty
    def text_msg(self) -> dict:
        msg = self.get_text_msg()
        return msg

    @lazyproperty
    def markdown_msg(self):
        return self.get_markdown_msg()

    @lazyproperty
    def get_jumpserver_dingtalk_msg(self) -> str:
        date_str = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
        msg = f"""## <font color="#FF0000">[JumpServer Monitoring Alert](https://jumpserver.example.com/)</font>🔥
### <font color="#FF0000">Alert Status</font>: {RiskLevelChoices.get_label(self.command['risk_level'])}
### <font color="#FF0000">Target Host</font>: {self.command['asset']}
### <font color="#FF0000">Target User</font>: {self.command['user']}
### <font color="#FF0000">Executed Command</font>: `{self.command['input']}`
### <font color="#FF0000">Alert Details</font>: User {self.command['user']} executed a risky command `{self.command['input']}` on {self.command['asset']}, please handle promptly!
### <font color="#FF0000">Trigger Time</font>: {date_str}"""
        return msg

    @lazyproperty
    def html_msg(self) -> dict:
        msg = self.get_html_msg()
        return msg

    @lazyproperty
    def html_msg_with_sign(self):
        msg = self.get_html_msg()
        msg['message'] = textwrap.dedent("""
        {}
        <small>
        <br />
        —
        <br />
        {}
        </small>
        """).format(msg['message'], self.signature)
        return msg

    @lazyproperty
    def text_msg_with_sign(self):
        msg = self.get_text_msg()
        msg['message'] = textwrap.dedent("""
        {}
        —
        {}
        """).format(msg['message'], self.signature)
        return msg

    @lazyproperty
    def signature(self):
        return get_login_title()

    # --------------------------------------------------------------
    # Support different messaging formats
    def get_dingtalk_msg(self) -> dict:
        # DingTalk restricts identical messages within a day, add timestamp suffix
        message = self.markdown_msg['message']
        time = local_now().strftime('%Y-%m-%d %H:%M:%S')
        suffix = '\n{}: {}'.format(_('Time'), time)

        return {
            'subject': self.markdown_msg['subject'],
            'message': message + suffix
        }

    def get_wecom_msg(self) -> dict:
        return self.markdown_msg

    def get_feishu_msg(self) -> dict:
        return self.markdown_msg

    def get_lark_msg(self) -> dict:
        return self.markdown_msg

    def get_email_msg(self) -> dict:
        return self.html_msg_with_sign

    def get_site_msg_msg(self) -> dict:
        return self.html_msg

    def get_slack_msg(self) -> dict:
        return self.markdown_msg

    def get_sms_msg(self) -> dict:
        return self.text_msg_with_sign

    @classmethod
    def get_all_sub_messages(cls):
        def get_subclasses(cls):
            """Returns all subclasses of argument, cls"""
            if issubclass(cls, type):
                subclasses = cls.__subclasses__(cls)
            else:
                subclasses = cls.__subclasses__()
            for subclass in subclasses:
                subclasses.extend(get_subclasses(subclass))
            return subclasses

        messages_cls = get_subclasses(cls)
        return messages_cls

    @classmethod
    def test_all_messages(cls, ding=True, wecom=False):
        messages_cls = cls.get_all_sub_messages()

        for _cls in messages_cls:
            try:
                _cls.send_test_msg(ding=ding, wecom=wecom)
            except NotImplementedError:
                continue


class SystemMessage(Message):
    def publish(self, is_async=False):
        subscription = SystemMsgSubscription.objects.get(
            message_type=self.get_message_type()
        )

        # Only send through enabled backends
        receive_backends = subscription.receive_backends
        receive_backends = BACKEND.filter_enable_backends(receive_backends)

        users = [
            *subscription.users.all(),
            *chain(*[g.users.all() for g in subscription.groups.all()])
        ]

        receive_user_ids = [u.id for u in users]
        backends_msg_mapper = self.get_backend_msg_mapper(receive_backends)
        if is_async:
            send_dingtalk_task(self.get_jumpserver_dingtalk_msg)
            # publish_task.delay(receive_user_ids, backends_msg_mapper)
        else:
            self.send_msg(receive_user_ids, backends_msg_mapper)

    @classmethod
    def post_insert_to_db(cls, subscription: SystemMsgSubscription):
        pass

    @classmethod
    def gen_test_msg(cls):
        raise NotImplementedError


class UserMessage(Message):
    user: User

    def __init__(self, user):
        self.user = user

    def publish(self, is_async=False):
        """
        Send messages through user-configured channels
        """
        sub = UserMsgSubscription.objects.get(user=self.user)
        backends_msg_mapper = self.get_backend_msg_mapper(sub.receive_backends)
        receive_user_ids = [self.user.id]
        if is_async:
            send_dingtalk_task(self.get_jumpserver_dingtalk_msg)
            # publish_task.delay(receive_user_ids, backends_msg_mapper)
        else:
            self.send_msg(receive_user_ids, backends_msg_mapper)

    @classmethod
    def get_test_user(cls):
        from users.models import User
        return User.objects.all().first()

    @classmethod
    def gen_test_msg(cls):
        raise NotImplementedError

Translated text here

values.yaml

    replicaCount: 1
    
    image:
      repository: nginx
      pullPolicy: IfNotPresent
      tag: ""
    
    imagePullSecrets: []
    nameOverride: ""
    fullnameOverride: ""
    
    serviceAccount:
      create: true
      annotations: {}
      name: ""
    
    podAnnotations: {}
    
    podSecurityContext: {}
    
    securityContext: {}
    
    service:
      type: ClusterIP
      port: 80
    
    ingress:
      enabled: false
    
    resources: {}
    
    autoscaling:
      enabled: false
    
    nodeSelector: {}
    
    tolerations: []
    
    affinity: {}

Modify the Image Version of the Jumpserver Core Module

To modify the image version of the Jumpserver core module, follow these steps:

# Step 1: Edit the core module deployment
kubectl -n jumpserver edit deployments.apps jms-k8s-jumpserver-jms-core

# Step 2: Locate the image field in the container configuration
# Find the containers section and modify the image version to:
# - name: jms-core
#   image: jumpserver/core:v4.6.1-ce

Kubernetes Lifecycle and Probes Deep Dive

Sat, 18 Feb 2023 00:00:00 +0000

Pod Lifecycle

📝 Note

Pods follow a predefined lifecycle, starting from the Pending phase. If at least one primary container starts successfully, the Pod transitions to the Running phase. Subsequently, depending on whether any container in the Pod exits with a failure status, it enters either the Succeeded or Failed phase.

When a Pod is deleted, some kubectl commands may display its status as Terminating. This Terminating state is not one of the official Pod phases.

Phase	Description
`Pending`	The Pod has been accepted by the Kubernetes system, but one or more containers have not been created or started. This phase includes time spent waiting for scheduling and downloading container images.
`Running`	The Pod is bound to a node, and all containers have been created. At least one container is still running, starting, or restarting.
`Succeeded`	All containers in the Pod have terminated successfully and will not be restarted.
`Failed`	All containers in the Pod have terminated, and at least one container exited due to failure (e.g., exited with a non-zero status or was terminated by the system without automatic restart configured).
`Unknown`	The Pod status cannot be retrieved, typically due to communication failures with the node hosting the Pod.

Container States

Kubernetes monitors the state of each container within a Pod, similar to how it tracks the Pod lifecycle.

Once the scheduler assigns a Pod to a node, the kubelet initiates container creation for the Pod through the container runtime. A container can be in one of three states: Waiting, Running, or Terminated. To inspect the state of containers in a Pod, use kubectl describe pod <pod-name>. The output includes the status of each container within the Pod.

Each state has specific implications:

`Waiting`

If a container is neither in the Running nor Terminated state, it is Waiting. A container in the Waiting state is still performing operations required to start successfully, such as pulling a container image from a registry or applying ConfigMap/Secret data to the container.

`Running`

The Running state indicates that the container is actively executing and functioning without issues.

`Terminated`

A container in the Terminated state has completed execution, either normally or due to a failure.

Pod Failure Scenarios

A Pod may encounter various exceptions during its lifecycle. Based on whether its containers are running, these failure scenarios can be broadly categorized into two groups:

Exceptions during container creation: These occur while the Pod is being scheduled or its containers are being created. The Pod remains stuck in the Pending phase.
Exceptions during container execution: These occur while containers are running. The Pod’s stage varies depending on the specific scenario.

4. Container Probes

📝 Note

Probes are a mechanism used by kubelet to periodically check the status of containers. To perform a check, kubelet can execute code inside the container or make a network request.

Liveness Probe

What is a Liveness Probe?

A liveness probe determines whether a container is running. If the probe fails, the kubelet kills the container, and the container is subjected to the restart policy.

If a container’s liveness probe fails multiple times, the kubelet will restart the container. If the liveness probe fails, the kubelet will kill the container, and the container will be subjected to the restart policy. If a container does not provide a liveness probe, the default status is Success.

The liveness probe does not wait for the readiness probe to succeed. If you want to wait for the liveness probe to execute before the readiness probe, you can define initialDelaySeconds, or use the startup probe.

When to Use a Liveness Probe?

If a container’s process can crash on its own, you may not need a liveness probe; kubelet will automatically restart the container based on the restartPolicy.

If you want the container to be killed and restarted when the probe fails, specify a liveness probe and set the restartPolicy to “Always” or “OnFailure”.

Readiness Probe

What is a Readiness Probe?

A readiness probe determines when a container is ready to accept traffic. This probe is useful when waiting for an application to perform time-consuming initial tasks, such as establishing network connections, loading files, and warming up caches.

If the readiness probe returns a failure status, Kubernetes removes the Pod from the endpoints of all associated Services.

The readiness probe continues to run throughout the container’s lifecycle.

When to Use a Readiness Probe?

Specify a readiness probe if you want to start sending traffic to a Pod only after the probe succeeds. In this case, the readiness probe might be the same as the liveness probe. However, the presence of a readiness probe in the specification ensures that the Pod does not receive any data during the startup phase and only starts receiving traffic after the probe succeeds.

You can also define a readiness probe if you want the container to enter a maintenance state independently. This probe should check a readiness-specific endpoint, which differs from the liveness probe.

If your application has strict dependencies on backend services, implement both liveness and readiness probes. After the liveness probe confirms the application is healthy, the readiness probe can perform additional checks to verify the availability of required backend services. This helps avoid directing traffic to Pods that might return errors.

For containers requiring large data loading, configuration file processing, or migrations during startup, use a startup probe. However, if you need to distinguish between a failed application and one still initializing, a readiness probe may be more appropriate.

Startup Probe (`startupProbe`)

What is a Startup Probe?

A startup probe checks whether an application within a container has started. It is designed for containers with slow startup times to prevent kubelet from terminating them prematurely before they begin running.

If configured, this probe disables liveness and readiness checks until the startup probe succeeds.

The startup probe executes only during the container’s initialization phase, unlike the readiness probe, which runs periodically.

When to Use a Startup Probe?

The startup probe is useful for Pods that contain containers requiring extended time to become ready. Instead of configuring a long liveness probe interval, you can set up a separate configuration option to probe the container during startup. This allows exceeding the duration permitted by the liveness check interval by a significant margin.

If a container’s startup time typically exceeds the total value of initialDelaySeconds + failureThreshold × periodSeconds, a startup probe should be configured to check the same endpoint used by the liveness probe. The default value for periodSeconds is 10 seconds. Set its failureThreshold high enough to ensure sufficient startup time for the container while retaining the default values for the liveness probe. This configuration helps mitigate deadlock scenarios.

Probe Execution

Probe	Target	Action	Effect	Runtime
Liveness	Container	Restart	Restart container	Entire container lifecycle
Readiness	Endpoint	Remove	Remove from service endpoints (no traffic)	Entire container lifecycle
Startup	Container	Restart	Restart container	Executed once after container starts Running

Probe Usage

There are four different methods to use probes to check containers. Each probe must be precisely defined as one of these four mechanisms:

Probe Type	Description
`exec`	Executes a specified command inside the container. The diagnosis is considered successful if the command exits with a status code of 0.
`tcpSocket`	Performs a TCP check on the specified port of the container’s IP address. The diagnosis is successful if the port is open. If the remote system (container) closes the connection immediately after opening it, this is still considered healthy.
`httpGet`	Sends an HTTP `GET` request to the specified port and path on the container’s IP address. The diagnosis is successful if the response status code is between 200 and 399 (inclusive).
`grpc`	Uses gRPC to perform a remote procedure call. The target should implement the gRPC health check. If the response status is “SERVING”, the diagnosis is considered successful.

Probe Parameters	Description
`initialDelaySeconds`	Number of seconds to wait after the container starts before initiating startup, liveness, and readiness probes. If a startup probe is defined, the delays for liveness and readiness probes will begin only after the startup probe succeeds. If `periodSeconds` is greater than `initialDelaySeconds`, `initialDelaySeconds` is ignored. The default is 0 seconds, and the minimum value is 0.
`periodSeconds`	Interval (in seconds) at which probes are executed. The default is 10 seconds. The minimum value is 1.

`timeoutSeconds`	The number of seconds to wait after a probe times out. Default is 1 second. Minimum value is 1.
`successThreshold`	Minimum consecutive successes required for a probe to be considered successful after failure. Default is 1. For liveness and startup probes, this value must be 1. Minimum value is 1.
`failureThreshold`	After `failureThreshold` consecutive failures, Kubernetes considers the overall check failed: container status becomes unready/unhealthy/inactive. Default is 3, minimum is 1. For liveness or startup probes: If ≥ `failureThreshold` probes fail, Kubernetes triggers container restart (following `terminationGracePeriodSeconds`). For readiness probes: kubelet continues executing failed probes but marks Pod’s `Ready` condition as false.
`terminationGracePeriodSeconds`	Configures the grace period for kubelet to wait between triggering container termination and forcing runtime stop. Default inherits Pod-level value (30s if unset). Minimum is 1. Added in Kubernetes v1.25, only effective for startup and liveness probes.

Example

apiVersion: v1
kind: Namespace
metadata:
  name: k8s-test


---
apiVersion: v1
kind: ConfigMap
metadata:
  name: probe-demo
  namespace: k8s-test
data:
  default.conf: |
    server {
        listen       80;
        server_name  localhost;
        keepalive_timeout  0;  # Disable Keep-Alive

        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }
    }


---
apiVersion: v1
kind: Service
metadata:
  name: probe-demo
  namespace: k8s-test
spec:
  selector:
    app: probe-demo
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 31080
  type: NodePort


---
apiVersion: v1
kind: Pod
metadata:
  name: probe-demo
  namespace: k8s-test
  labels:
    app: probe-demo
spec:
  restartPolicy: OnFailure  # Pod restart policy: OnFailure (restart only on non-zero exit code), Always (restart on any exit), Never (no restart)
  containers:
    - name: probe-demo
      image: 192.168.142.99:7891/devops/nginx:latest
      # command: ["/bin/sh", "-c", "sleep 10"]  # Simulate Pod Succeeded status
      # command: ["/bin/sh", "-c", "sleep infinity"]  # For startup probe testing
      command: ["/bin/sh", "-c", "set -ex && nohup nginx && touch /tmp/healthy && sleep 30 && rm -f /tmp/healthy && sleep 600"]
      # startupProbe:
      #   exec:
      #     command:
      #       - cat
      #       - /tmp/healthy
      #   initialDelaySeconds: 5  # Wait 5s after container starts
      #   periodSeconds: 5       # Probe every 5s
      #   timeoutSeconds: 1      # Probe timeout
      #   successThreshold: 1    # Minimum consecutive successes to consider probe successful
      #   failureThreshold: 3    # Consecutive failures needed to mark probe failed
      # livenessProbe:
      #   exec:
      #     command:
      #       - cat
      #       - /tmp/healthy
      #   initialDelaySeconds: 5
      #   periodSeconds: 5
      #   timeoutSeconds: 1
      #   failureThreshold: 3
      readinessProbe:
        exec:
          command:
            - cat
            - /tmp/healthy
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 1
        failureThreshold: 3
      
      volumeMounts:
        - name: config-volume
          mountPath: /etc/nginx/conf.d/default.conf
          subPath: default.conf

  volumes:
    - name: config-volume
      configMap:
        name: probe-demo
        items:
          - key: default.conf
            path: default.conf

Kubernetes on Kang's Blog

Ascend NPU Deployment Guide for Kubernetes

Ascend NPU Deployment Guide for Kubernetes

Overview

Preparation

Create Node Labels

Create System User

Create Log Directory

Install Ascend Docker Runtime

Download Installation Package

Installation Steps

Step 1: Enter Installation Package Directory

Step 2: Verify Package Integrity

Step 3: Add Executable Permission

Step 4: Execute Installation

Configure Containerd

Configuration Method 1: Cgroup v1

Configuration Method 2: Cgroup v2

Restart Services

Verify Installation

Deploy Ascend Device Plugin

Prepare Images

Download Deployment Configuration Files

Choose the Appropriate YAML File

Deploy Device Plugin

Verify Deployment

Using NPU Compute Cards

Pod Configuration Example

Kubernetes Package Management: The Ultimate Helm Guide

Introduction

Helm’s Workflow

Three Core Concepts

Chart

Repository

Release

Installation

Create Project Templates

Summary

Guide to Configuring TCP Service Forwarding with Ingress-Nginx in Kubernetes

Prerequisites

Configuration Steps

Configure TCP Service Mapping

Update Ingress Controller Configuration

Configuring Ingress Controller Service

Verification

Notes

Kubernetes Ingress-NGINX Service Certificate Configuration

Service Deployment

Deploy MetalLB

Deploy ingress-nginx

Configuring Certificates

Manual Self-Signed Certificate

Manual Trusted Certificate Application

Using cert-manager to Obtain Certificates

Installing cert-manager

Configuring SelfSigned Issuer Type Certificates

Configuring ACME Issuer with HTTP01 Certificate Type

Configuring ACME Issuer dns01-type Certificates

Deploying JumpServer on Kubernetes with DingTalk Alert Integration

Ingress Configuration

JumpServer Deployment

Core Component Image

values.yaml

Modify the Image Version of the Jumpserver Core Module

Kubernetes Lifecycle and Probes Deep Dive

Pod Lifecycle

Container States

Waiting

Running

Terminated

Pod Failure Scenarios

4. Container Probes

Liveness Probe

What is a Liveness Probe?

When to Use a Liveness Probe?

Readiness Probe

What is a Readiness Probe?

When to Use a Readiness Probe?

Startup Probe (startupProbe)

What is a Startup Probe?

`Waiting`

`Running`

`Terminated`

Startup Probe (`startupProbe`)