OpenNJet KIC V2.0 released!

[Shenzhen] Yuanchuanghui: 5.26pm, the party hall is waiting for you.”

NGINX evolves to cloud native, All in OpenNJet

Overview

OpenNJet KIC (Kubernetes Ingress Controller) is based on the dynamic characteristics and high-performance implementation of OpenNJet proxy. Make up for the shortcomings of nginx in cloud native scenarios. Provides rich traffic management capabilities, such as dynamic location, host/path routing, load balancing, dynamic upstream, canary publishing, TLS Termination/SNI, TCP/UDP, WebSocket, etc.

Main features of this version:

Supports sharding processing Ingress/VS CR
Support integration with ADC
Support HTTP header operations
Support TCP proxy
Support cross-namespace
Support WebSocket proxy
Support UDP proxy
Support dynamic NJet VS/Accesslog
Support TCP active health check
Support dynamic adjustment of the number of worker processes

This article provides a brief introduction to these new features.

The architecture diagram is as follows:

New features overview

Sharding processing Ingress/VS CR

There are special KICs for processing different Ingress/VS resources. This is the KIC sharding mechanism.

Different Ingress/VS resources are identified by ingressClass. Ingress resources can be identified by the annotation kubernetes.io/ingress.class or spec.ingressClassName, and VS resources can be identified by spec.ingressClassName.

Note that each KIC in this sharding mechanism is called a type of KIC and corresponds to IngressClass one-to-one. In a k8s cluster, if there are multiple Njet KICs, multiple IngressClass objects need to be created. Each type of KIC can have multiple copies (corresponding to multiple pods in the k8s architecture).

The sharding mechanism causes each KIC to have its own Ingress/VS resources that it is interested in, rather than all the Ingress/VS resources in the k8s cluster. It aims to solve the problem that a single type of KIC cannot withstand the pressure of full configuration.

Integrate with ADC

After KIC is integrated with ADC, ADC can serve as KIC's front-end LB, managing client traffic through ADC and ultimately load balancing to KIC services. KIC has completed the following functions:

ADC domain name registration: KIC registers with ADC the domain names of services in the k8s cluster managed by KIC, such as services managed through k8s ingress and vs CR. This function allows clients to make requests directly through domain names (the DNS server of the ADC needs to be configured as the default DNS server).
ADC SlbPool registration: KIC registers the application pool (nodeIP+nodePort) associated with the KIC service with ADC. This function allows ADC to route to the KIC service.
ADC VS registration: KIC registers a VS with ADC and associates it with the application pool created in the second step. This function allows the client to directly access the VS, so that ADC can serve as the front-end LB of KIC.

The VIP in ADC VS corresponds to the domain name of the service managed by KIC.
The VIP in ADC VS is a public network IP, which can be directly accessed by external clients.

Overall structure

Scene graph:

Interaction architecture diagram:

HTTP header manipulation

OpenNJet KIC header operation can modify the client request header and operate on the response header of the proxied service. Including user-defined headers and headers defined in the HTTP specification.

TCP proxy

KIC TCP proxy can proxy upstream TCP services. TCP proxy provides load balancing capabilities. The TCP proxy is fully dynamically implemented and does not require OpenNJet to perform a reload operation.

The proxied TCP service will have its own listening port. The more proxied services, the more ports that need to be listened. In the traditional Nginx implementation, each new listen requires reload configuration. In order to solve the reload problem, we use port forwarding. The method implements the proxy of TCP service. The specific design is as follows:

OpenNJet stream configuration pre-listens a 12003 port (the port monitored by the proxy TCP service will be redirected to 12003).
KIC generates iptables rules through the customer-defined API, and redirects the requested TCP service port to port 12003 through the iptables rules.
The "TCP service port" that OpenNJet stream accesses through the client matches the upstream configured by the customer in advance through the API. This process is matched through the stream map.
OpenNJet stream sends the client request to the server in the upstream that successfully matches (the load balancing is implemented by stream lua).

iptables rules and map content updates are dynamically updated using API, avoiding reload operations.

OpenNJet configuration is implemented as follows:

 map $njtmesh_port \$stream_upstream {
 }
 
 server {
 listen 12003 mesh;
 set $proxy_upstream_name \$stream_upstream;
 proxy_pass upstream_balancer;
 }

The port requested by the client is always the port of the listener in TransportServer.
The port cannot be used by the proxy service because it will be used internally by KIC. The description is described in the following table:

port	illustrate
80	http proxy
443	https proxy
12001	OpenNJet control plane
12002	tcp lua upstream
12003	TCP proxy port
12004	UDP proxy port
8080	stub_status and http lua upstream
8081	KIC pod readiness-port

Across namespaces

The built-in Ingress of K8s can only handle routing of services in the same ns, and cannot perform cross-ns operations. The cross-namespace configuration feature resolves this limitation.

In addition to Ingress supporting cross-namespace configuration, VS also supports cross-ns configuration. Users can choose according to their own circumstances.

Ingress implements this function through different Ingress types. Ingress itself has no concept of type. We express it by adding the annotation "njet.org.cn/mergeable-ingress-type". Annotations can take values from the following table:

value	illustrate	Remark
master	Main Ingress	one
minion	FromIngress	Can be multiple, associated with master Ingress through host

The master and minion of the same host can be in the same ns or different ns.

In addition to cross-ns configuration, you can choose this function. When a host has a large number of paths and the maintenance of a single Ingress is complicated, you can also choose this function to manage paths.

VS CR has the same function as Ingress, but the implementation method is different. VS CR performs cross-ns configuration by referencing a sub-routing resource. The sub-route is a string in ns/name format, which is used to represent VSR resources. VSR is a K8s CR, a new CR introduced for this feature.

WebSocket proxy

WebSocket is an independent TCP-based application layer protocol and a full-duplex communication protocol. Its only relationship with HTTP is that its handshake is interpreted by the HTTP server as an HTTP upgrade request. When the handshake ends, it has nothing to do with HTTP. The protocol has two parts: handshake and data transfer. The handshake phase is shown in the figure below:

Ingress implements this function by adding annotations, which we express by adding the annotation "njet.org.cn/websocket-services".

Annotation name	value format	describe
njet.org.cn/websocket-services	service,service2 (comma separated service names)	Websocket support

VS does not require any configuration.

UDP proxy

KIC UDP proxy can proxy upstream UDP services. UDP proxy provides load balancing capabilities. The UDP proxy is fully dynamically implemented and does not require OpenNJet to perform a reload operation.

The UDP service proxy implementation is basically the same as the TCP service proxy, except that the OpenNJet stream configuration pre-listens to a 12004 port (the port monitored by the proxy UDP service will be redirected to 12004), while TCP pre-listens to a 12003 port. For detailed instructions, refer to TCP Proxy.

OpenNJet configuration is implemented as follows:

map $njtmesh_port \$stream_upstream_udp {
}

server {
listen 12004 udp mesh;
set $proxy_upstream_name $stream_upstream_udp;
proxy_pass upstream_balancer;
}

Dynamic NJet VS

Compared with the first version, we changed the single-server and multiple-location approach to achieve HTTP host header matching and path matching to the multiple-server and multiple-location approach. We used the nginx standard server_name to implement host header matching, but dynamically updated the configuration. Yes, no reload operation is required. The second version does not add additional features for Ingress and VirtualServer.

DynamicAccesslog

The KIC Accesslog function allows you to apply accesslog policies (including switch status and accesslog file path) individually to a certain application. Global settings can also be made through njet-config ConfigMap, but "applying the accesslog policy individually to an application" has a higher priority.

TCP proactive health check

For Transport Server resources, it is supported to configure a TCP port. The active health check will detect whether the TCP port is listening normally according to the configured check interval, and the backend will be online and offline based on the check results.

apiVersion: k8s.njet.org/v1alpha1
 kind: TransportServer
 metadata:
 name: testapp-tcp
 spec:
 listener:
 name: test-tcp
 protocol: TCP
 upstreams:
 - name: testapp1
 service: testapp
 port: 80
 healthCheck:
 enable: true
 interval: 20s
 timeout: 5s
 fails: 1
 passes: 1
 port: 83
 action:
 pass: testapp1

The configuration instructions are as follows:

Field	describe	type	Is it required?
enable	Whether to enable health check, default false	boolean	No
interval	The interval between checks. Default is 5	string	No
fails	After a few failures, you will be considered offline. Default 1 time	integer	No
passes	It will be considered online after several successes. Default 1 time	integer	No
port	The port where the health check service is located	integer	No
timeout	Health check connection timeout	string	No

Dynamic worker process number adjustment

Supports configuring the number of worker processes of njet instances through ConfigMap resources. After the configuration items in ConfigMap are updated, the number of worker processes will be dynamically modified.

Configuration items	illustrate	Field Type	default value	Remark
worker-processes	Number of Worker processes (allowed values 1 - 512)	String	The default is the number of processes automatically generated by auto based on the number of cpu cores.

Reference link

OpenNJetKIC User Manual

OpenNJet KIC source code address

The NJet application engine achieves unique runtime dynamic configuration loading capabilities through kernel reconstruction and is a new generation of high-performance Web application engine . NJet has high-performance data plane processing capabilities and schedules multiple auxiliary functions such as clustering, high availability, active health checks, and declarative APIs through NJet's unique co-pilot CoPilot service framework to facilitate function expansion and isolate management/control function pairs. Due to the impact on the data plane, the performance of the NJet application engine exceeds three times that of the Envoy application engine recommended by CNCF. Mail group official website