Author: Wang Jinshan, Microservice Architect of the Technology Department of Beijing Yunsichangxiang Technology Co., Ltd., responsible for the research and development of the company's API gateway and service grid
Speed cloud introduction
Founded in October 2014, Speed Cloud is committed to helping enterprises achieve digital transformation through cloud-native technology. It has two core product systems, cloud-native application platform TCAP and cloud-native data platform KubeData. The products include cloud-native DevOps, container cloud PaaS, and intermediate Software services, edge computing, microservice governance, service mesh, API gateway, etc.
demand background
There are a variety of gateway software in the PaaS platform of Speed Cloud:
- Traffic gateway based on HAProxy and Nginx Ingress
- Spring Cloud Gateway of Spring Cloud Microservice System
- Kong as API Gateway
- Istio Ingress Gateway under the service mesh system
Although the above products each have their own application scenarios, the problem they bring is that the technology stacks are different. After introducing multiple tools to meet different needs, it will bring about a sharp increase in maintenance costs. Therefore, finding a tool that can meet all needs and use a unified technology stack has become our goal. Higress fit our needs exactly.
Higress solution
Alternative to Nginx Ingress
Higress can be used as the Ingress entry gateway of the K8s cluster, and is compatible with a large number of K8s Nginx Ingress annotations, and can quickly and smoothly migrate from K8s Nginx Ingress to Higress. The following is an example of implementing REST routing based on Higress' own annotations and compatible with Nginx Ingress annotation rewriting paths:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# 兼容 Nginx Ingress 注解
nginx.ingress.kubernetes.io/rewrite-target: /
# Higress 注解，支持 method/header/query 匹配路由
higress.io/match-method: POST
higress.io/exact-match-query-higressQuery: hi
higress.io/prefix-match-header-x-higress-header: hi
name: foo
spec:
ingressClassName: higress
rules:
- host: foo.example.com
http:
paths:
- pathType: Prefix
path: /foo
backend:
service:
name: foo-service
port:
number: 5678
And the Lua code performance of Nginx Ingress is relatively poor, and the performance of Higress compared with Nginx Ingress is greatly improved, as shown in the following figure:
Alternative to Spring Cloud Gateway
Under the Spring Cloud microservice system, as a microservice gateway, it must be connected with the microservice registry to realize service discovery. Higress provides the CRD McpBridge, which can be easily connected to various registration centers. The Spring Cloud registration center we use is Nacos, and the configuration of McpBridge is as follows:
apiVersion: networking.higress.io/v1
kind: McpBridge
metadata:
name: default
namespace: higress-system
spec:
registries:
# 定义一个名为 my-nacos 的服务来源
- name: my-nacos
# 注册中心类型是 Nacos 2.x，支持 gRPC 协议
type: nacos2
# 注册中心的访问地址，可以是域名或者IP
domain: 127.0.0.1
# 注册中心的访问端口，Nacos 默认都是 8848
port: 8848
# Nacos 命名空间 ID
nacosNamespaceId: d8ac64f3-xxxx-xxxx-xxxx-47a814ecf358
# Nacos 服务分组
nacosGroups:
- DEFAULT_GROUP
Next, configure Ingress to forward to the service user-center registered on Nacos:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
higress.io/destination: user-center.DEFAULT-GROUP.d8ac64f3-xxxx-xxxx-xxxx-47a814ecf358.nacos
name: user
namespace: default
spec:
rules:
- http:
paths:
- backend:
resource:
apiGroup: networking.higress.io
kind: McpBridge
name: default
path: /
pathType: Prefix
In this way, Spring Cloud microservices can be connected to the Higress gateway without any modification. Compared with traditional Java microservice gateways such as Spring Cloud Gateway/Zuul, the performance of Higress is more than 2 times higher, which can significantly reduce resource costs.
Replace Kong
The API gateway product of Speed Cloud is used to help enterprises and developers complete the management of the entire life cycle of API creation, maintenance, release, monitoring and alarming. Through the API gateway, the back-end service is opened in the form of API and provided to all parties. Authentication and authentication capabilities are the key capabilities in the API gateway. Our API gateway was originally built based on Kong, mainly using Kong's Key Auth/Basic Auth/JWT Auth/HMAC Auth plug-ins, and Higress also provides these plug-ins ability:
Taking Key Auth as an example, the WasmPlugin CRD provided by Higress can be configured as follows:
apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
name: key-auth
namespace: higress-system
spec:
# 全局配置，配置认证规则
defaultConfig:
consumers:
- credential: 2bda943c-xxxx-xxxx-xxxx-00163e1250b5
name: consumer1
- credential: c8c8e9ca-xxxx-xxxx-xxxx-e700dcc40e35
name: consumer2
keys:
- x-api-key
# 从请求header识别key
in_header: true
# 开启全局认证，consumer未识别将拒绝访问
global_auth: true
# 匹配规则，配置授权规则
matchRules:
# 路由级生效配置，匹配default命名空间下名为foo的ingress
- ingress:
- default/foo
config:
# 仅允许 consumer1 访问
allow:
- consumer1
# 域名级生效配置
- domain:
- www.test.com
- *.example.com
config:
# 仅允许 consumer1 访问
allow:
- consumer2
url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/key-auth:1.0.0
Initiate an authentication request, because xxx.exmaple.com only authorizes consumer2 access, so the following curl command will return 403:
curl http://xxx.example.com/test -H 'x-api-key: 2bda943c-xxxx-xxxx-xxxx-00163e1250b5'
In addition, Higress provides a more flexible custom plug-in mechanism. Compared with Kong’s new plug-ins, which need to redeploy the gateway, Higress can dynamically expand and hot-update the plug-in logic, which is completely non-destructive to traffic, and can also support multi-language development without limitation. in the Lua language.
Alternative to Istio Ingress Gateway
The service grid product of Speed Cloud is built based on Istio. In the Istio service grid architecture, the Istio Ingress Gateway is generally responsible for north-south traffic management. Because Higress can also support Istio API, we also use Higress to unify the north-south traffic management in the service mesh.
Higress itself does not have a strong dependence on Istio, so the API support of Istio is disabled by default, which needs to be enabled through helm parameter configuration:
helm upgrade higress -n higress-system higress.io/higress --reuse-values --set global.enableIstioAPI=true
Once enabled, you can directly use the Istio API to manage routes on Higress:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: devops
namespace: higress-system
spec:
selector:
higress: higress-system-higress-gateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- devops.com
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: devops
namespace: higress-system
spec:
gateways:
- higress-system/devops
hosts:
- devops.com
http:
- name: default
route:
- destination:
host: devops.default.svc.cluster.local
Based on the Istio API, Higress also supports TCP routing, which can replace our previous use of HAProxy to proxy MySql and other middleware functions, for example:
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: mysql
namespace: higress-system
spec:
selector:
higress: higress-system-higress-gateway
servers:
- hosts:
- '*'
port:
name: tcp
number: 3306
protocol: TCP
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: mysql
namespace: higress-system
spec:
gateways:
- mysql
hosts:
- '*'
tcp:
- match:
- port: 3306
route:
- destination:
host: mysql
port:
number: 3306
subset: v1
Earnings and Outlook
The migrated architecture is based on the Higress integration of various technology stacks at the product level, which enhances the user experience. The specific benefits are as follows:
- Higress supports K8s Ingress API and Istio Gateway/VirtualService API at the same time, and multiple clusters can be upgraded quickly and smoothly
- Using Higress to unify multiple gateways, unify the technology stack of traffic entry/routing load/security authentication, and realize cost reduction and efficiency increase
- Higress is based on Envoy and uses the same technology stack as Sidecar for east-west traffic management, which reduces the cost of expansion and maintenance for developers
Finally, we also put forward some expectations for the future development of Higress:
- Hope to support Gateway API as soon as possible, and support TCPRoute/UDPRoute to enhance the four-layer capability
- Looking forward to the community's Wasm plug-in ecology becoming more and more abundant, providing more out-of-the-box capabilities
- At present, Speed Cloud will deploy multiple sets of Higress gateways in a K8s cluster, hoping to provide an Operator mechanism to simplify operation and maintenance