Perhaps Higress is the best cloud native gateway out there?

Higress is a cloud-native API gateway based on Alibaba's internal Envoy Gateway practice and built with open source Istio + Envoy as the core. It realizes the high integration capability of traffic gateway + microservice gateway + security gateway, and deeply integrates Dubbo, Nacos, Microservice technology stacks such as Sentinel can help users greatly reduce gateway deployment and operation and maintenance costs; it fully supports Ingress and Gateway APIs in terms of standards, and actively embraces standard API specifications under cloud native; at the same time, Higress Controller also supports Nginx Ingress Smooth migration helps users quickly migrate to Higress at zero cost.

In the industry, gateways are usually divided into two categories: traffic gateways and business gateways. Traffic gateways mainly provide global policy configurations that have nothing to do with back-end services. For example, Alibaba's internal unified access gateway Tengine is a typical traffic gateway. ; As the name suggests, the business gateway mainly provides independent business domain level and tightly coupled policy configuration with the back-end business. As the application architecture model evolves from a single entity to the current distributed microservices, the business gateway also has a new name - microservice gateway.

Under the microservice architecture in the virtualization period, businesses usually adopt a two-layer architecture of traffic gateway + microservice gateway. The traffic gateway is responsible for north-south traffic scheduling and security protection, and the microservice gateway is responsible for east-west traffic scheduling and service governance. In the container In the cloud-native era dominated by K8s, Ingress has become the gateway standard of the K8s ecosystem, giving the gateway a new mission, making it possible to combine traffic gateway + microservice gateway into one.

As a public network gateway facing north and south, it is a common requirement to use Waf to protect abnormal traffic. And as the Internet environment becomes more and more complex, users' demands for protection continue to increase. The common practice is to connect the traffic first Waf security gateway, after filtering, forwards the traffic to the traffic gateway, and finally reaches the microservice gateway; Higress hopes that through the built-in Waf module, the user's request link can complete Waf protection, traffic distribution, and microservice governance at the same time only through Higress. It can reduce link RT and reduce gateway operation and maintenance complexity. Therefore, Higress achieves the high integration capability of traffic gateway + microservice gateway + security gateway.

scenes to be used

Higress supports a variety of features and is suitable for a variety of scenarios:

• Kubernetes Ingress gateway:
  Higress can be used as the Ingress entry gateway of the K8s cluster, and is compatible with a large number of K8s Nginx Ingress annotations, and can be quickly and smoothly migrated from K8s Nginx Ingress to Higress.
  Supports the Gateway API standard and supports smooth migration of users from the Ingress API to the Gateway API.
• 微服务网关:
  Higress 可以作为微服务网关, 能够对接多种类型的注册中心发现服务配置路由，例如 Nacos, ZooKeeper, Consul, Eureka 等。
  并且深度集成了 Dubbo, Nacos, Sentinel 等微服务技术栈，基于 Envoy C++ 网关内核的出色性能，相比传统 Java 类微服务网关，可以显著降低资源使用率，减少成本。
• Security protection gateway:
  Higress can be used as a security protection gateway, providing WAF capabilities and supporting multiple authentication strategies, such as key-auth, hmac-auth, jwt-auth, basic-auth, oidc, etc.

core advantages

• The production level
  is derived from Alibaba's internal products that have been produced and verified for more than 2 years, and supports large-scale scenarios with hundreds of thousands of requests per second.
  Completely get rid of traffic jitter caused by reload, configuration changes take effect in milliseconds and the business is not affected.
• Smooth evolution
  supports multiple registration centers such as Nacos/Zookeeper/Eureka, can perform service discovery without relying on K8s Service, and supports smooth evolution from non-container architecture to cloud-native architecture.
  Supports smooth migration from Nginx Ingress Controller, supports smooth transition to Gateway API, and supports smooth evolution of business architecture to ServiceMesh.
• Strong compatibility:
  It is compatible with 80%+ usage scenarios of Nginx Ingress Annotation, and provides Higress Annotation annotations with richer functions.
  Compatible with Ingress API/Gateway API/Istio API, multiple CRDs can be combined to achieve refined traffic management.
• It provides three plug-in extension mechanisms: Wasm, Lua, and out-of-process for easy expansion
  . It supports plug-ins written in multiple languages. The effective granularity supports global level, domain name level, and routing level.
  Plug-ins support hot updates, and changes to plug-in logic and configuration will have no impact on traffic.

Architecture

Overall, the Higress gateway consists of the control plane component higress-controller and the data plane component higress-gateway. higress-gateway is responsible for carrying data traffic, and higress-controller is responsible for managing configuration distribution.

Higress

The data plane component higress-gateway is a gateway component developed based on Envoy. It is responsible for receiving and processing traffic. It supports HTTP/1.1, HTTP/2, gRPC and other protocols, and supports TLS, mTLS, WAF, current limiting, circuit breaker, retry, and load balancing. , routing, forwarding, redirection, cross-domain and other functions, which means that the real traffic processing is completed in higress-gateway.

The control plane component higress-controller is responsible for managing and delivering configurations. It supports Ingress API, Gateway API, Istio API, multiple registration centers, multiple authentication strategies, multiple plug-in extension mechanisms, and multiple CRDs to achieve traffic refinement. Management, that is to say, all configurations are delivered to higress-gateway through higress-controller.

Install

The installation of Higress is very simple. You only need to install it through Helm. If you want to customize it according to your own needs, you can do it by modifying the parameters of Helm. For complete parameter introduction, you can view the operation and maintenance parameter description. Some commonly used customizable parameters are as follows Shown:

By default, ingress-gateway will be exposed through a LoadBalancer Service. If your cluster does not support LoadBalancer type Service, you can modify the Service type by modifying the higress-core.gateway.service.type parameter, or by modifying higress-core. gateway.hostNetwork parameter to directly use the host network. For example, we use hostNetwork here to directly use the host network:

helm repo add higress.io https://higress.io/helm-charts
helm upgrade --install higress -n higress-system higress.io/higress --set higress-core.gateway.hostNetwork=true,higress-core.gateway.service.type=ClusterIP --create-namespace
1.
2.

In addition, Higress also supports Istio CRD (optional), but the Istio CRD needs to be installed in advance in the cluster. If you do not want to install Istio, you can also only install the Istio CRD:

helm repo add istio https://istio-release.storage.googleapis.com/charts
helm install istio-base istio/base -n istio-system --create-namespace
1.
2.

In this mode, you need to update the deployment parameters of Higress:

helm upgrade higress -n higress-system --set global.enableIstioAPI=true higress.io/higress --reuse-values
1.

Higress also supports Gateway API CRD (optional). Similarly, we also need to install the CRD of Gateway API in advance: https://github.com/kubernetes-sigs/gateway-api/releases. For example, we install version 1.0.0 here. CRD, the specific version can be selected according to the actual situation:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
1.

In this mode, you need to update the deployment parameters of Higress:

helm upgrade higress -n higress-system --set global.enableGatewayAPI=true higress.io/higress --reuse-values
1.

After the installation is complete, we can check the deployment status of Higress:

$ kubectl get pods -n higress-system
NAME                                 READY   STATUS    RESTARTS   AGE
higress-console-796544b9df-9v5hn     1/1     Running   0          4m57s
higress-controller-8d4bb7456-4nx6l   2/2     Running   0          4m57s
higress-gateway-596f4f6d9-d46wg      1/1     Running   0          4m57s
higress-gateway-596f4f6d9-mbhp9      1/1     Running   0          4m57s
$ kubectl get svc -n higress-system
NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                                                    AGE
higress-console      ClusterIP   10.96.2.141   <none>        8080/TCP                                                   9m35s
higress-controller   ClusterIP   10.96.0.98    <none>        8888/TCP,15051/TCP,15010/TCP,15012/TCP,443/TCP,15014/TCP   9m35s
higress-gateway      ClusterIP   10.96.3.17    <none>        80/TCP,443/TCP                                             9m35s
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

Next we can create an Ingress object as shown below to expose the service: higress-console 

# higress-console.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: higress-console
  namespace: higress-system
spec:
  ingressClassName: higress
  rules:
    - host: higress.k8s.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: higress-console
                port:
                  number: 8080
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

Just apply the above Ingress object directly:

$ kubectl apply -f higress-console.yaml
$ kubectl get ingress -n higress-system
NAME              CLASS     HOSTS               ADDRESS   PORTS   AGE
higress-console   higress   higress.k8s.local             80      15s
1.
2.
3.
4.

Since we have set higress-gateway to hostNetwork mode here, we only need to parse higress.k8s.local to any node where higress-gateway is located, so that we can access higress- through higress.k8s.local console service.

Initialize administrator

Basic use

When we access higress-console for the first time, we need to initialize the administrator account. After the initialization is completed, we can log in to the higress-console console through the administrator account.

higress console

Then we can configure our gateway through the higress-console console, including creating routing rules, services, certificates, etc.

For example, there is a foo service as shown below under the default namespace:

# foo.yaml
kind: Pod
apiVersion: v1
metadata:
  name: foo-app
  labels:
    app: foo
spec:
  containers:
    - name: foo-app
      image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/http-echo:0.2.4-alpine
      args:
        - "-text=foo"
---
kind: Service
apiVersion: v1
metadata:
  name: foo-service
spec:
  selector:
    app: foo
  ports:
    # Default port used by the image
    - port: 5678
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.

Now we want to create a route corresponding to http://foo.bar.com/foo to point to this service. Here we have two ways to achieve it, one is to create it through the higress-console console, and the other is to create it through a YAML file.

Create via console

First click the domain name management navigation bar on the left, then click the Create Domain Name button on the right side of the page, fill in the form as shown in the figure below and click the OK button.

Create domain name

Then click the routing management navigation bar on the left, then click the Create Route button on the right side of the page, fill in the form as shown in the picture below and click the OK button (pay attention to selecting the target service).

Create route

Next, we also only need to resolve the domain name foo.bar.com to any node where the higress-gateway is located, so that we can access the foo-service service through foo.bar.com.

$ curl http://foo.bar.com/foo
foo
1.
2.

Created from YAML files

Similarly, we can also create routes through YAML files (actually an Ingress object). First we need to create a file, and then use the YAML below to create the routing configuration we need. foo-route.yaml 

# foo-route.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foo
spec:
  ingressClassName: higress
  rules:
    - host: foo.bar.com
      http:
        paths:
          - pathType: Prefix
            path: "/foo"
            backend:
              service:
                name: foo-service
                port:
                  number: 5678
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.

Then we only need to apply the above YAML file (remember to delete the domain name and route added in the console first):

$ kubectl apply -f foo-route.yaml
$ kubectl get ingress
NAME   CLASS     HOSTS         ADDRESS   PORTS   AGE
foo    higress   foo.bar.com             80      4s
1.
2.
3.
4.

Also now we can access the foo-service service through foo.bar.com.

$ curl foo.bar.com/foo
foo
1.
2.

However, it should be noted that the routing configuration managed through YAML files will not be synchronized to the console, so you need to pay attention when managing routing.

Advanced usage

Standard K8s Ingress resources can only handle HTTP(S) traffic routing in simple scenarios, and cannot handle issues such as traffic segmentation, timeout retry, header control, and cross-domain issues. Therefore, different Ingress Controllers use customized Ingress Annotation to enhance Ingress capabilities. The common Nginx Ingress Controller introduces more than 100 Annotations to extend Ingress's traffic management and security protection. Currently, Higress is fully compatible with most Nginx Ingress Annotations, making it easy for users to seamlessly migrate from Nginx Ingress to Higress, so we can continue to use Nginx Ingress's Annotations to configure Higress.

We can continue to use the Annotation prefix nginx.ingress.kubernetes.io of Nginx Ingress according to our usage habits, or use the Annotation prefix higress.io of Higress Ingress. The two are equivalent. Let's demonstrate it through a few simple examples. How to use Annotation to configure Higress.

Cross domain

Cross-domain resource sharing CORS refers to allowing web application servers to perform cross-domain access control to achieve secure cross-domain data transmission. We can configure Higress's cross-domain policy through the following Annotation:

• higress.io/enable-cors: true or false, used to enable or disable cross-domain.
• higress.io/cors-allow-origin: Allowed third-party sites, supports generic domain names, separated by commas; supports wildcards. The default value is *, which allows all third-party sites.
• higress.io/cors-allow-methods: Allowed request methods, such as GET and POST, separated by commas; supports wildcard *. The default values ​​are GET, PUT, POST, DELETE, PATCH, OPTIONS.
• higress.io/cors-allow-headers: Allowed request headers, comma separated; supports wildcard *. The default value is DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization.
• higress.io/cors-expose-headers: Allowed response headers, comma separated.
• higress.io/cors-allow-credentials: true or false. Whether to allow the carrying of credential information. Allowed by default.
• higress.io/cors-max-age: The maximum cache time of preflight results, in seconds; the default value is 1728000.

For example, we now have a requirement that cross-domain requests are restricted to requests that can only come from the example.com domain, and the HTTP methods can only be GET and POST. The allowed request header is X-Foo-Bar, and credential information is not allowed. Then we can configure Higress through the following Annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/enable-cors: "true"
    higress.io/cors-allow-origin: "example.com"
    higress.io/cors-allow-methods: "GET,POST"
    higress.io/cors-allow-headers: "X-Foo-Bar"
    higress.io/cors-allow-credentials: "false"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /hello
            pathType: Exact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.

Rewrite

Before the request is forwarded to the target backend service, Rewrite can modify the path (Path) and host domain (Host) of the original request. We can configure Higress's Rewrite rewrite strategy through the following Annotation:

• higress.io/rewrite-target: Rewrite Path and support Capture Group.
• higress.io/upstream-vhost: Rewrite Host.

Rewrite rewrite Path

Rewrite request example.com/test to example.com/dev before forwarding to the backend service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/rewrite-target: "/dev"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /test
            pathType: Exact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

Remove the Path prefix /v1 before forwarding the request to example.com/v1/app to the backend service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/rewrite-target: "/$2"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /v1(/|$)(.*)
            pathType: ImplementationSpecific
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

Change the Path prefix /v1 to /v2 before forwarding the request to example.com/v1/app to the backend service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/rewrite-target: "/v2/$2"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /v1(/|$)(.*)
            pathType: ImplementationSpecific
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

Rewrite rewrite Host

Rewrite the request example.com/test to test.com/test before forwarding it to the backend service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/upstream-vhost: "test.com"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /test
            pathType: Exact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

Redirect

Redirecting changes the original client request to a target request.

Configure HTTP redirect to HTTPS

We can use the following Annotation to configure Higress to redirect HTTP requests to HTTPS:

• higress.io/ssl-redirect: HTTP redirect to HTTPS
• higress.io/force-ssl-redirect: HTTP redirect to HTTPS

Higress does not treat the above two annotations differently, and both forcefully redirect HTTP to HTTPS.

For example, if we need to redirect the request http://example.com/test to https://example.com/test, we can configure Higress through the following Annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/ssl-redirect: "true"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /test
            pathType: Exact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

permanent redirect

• higress.io/permanent-redirect: The target URL of permanent redirection must contain scheme (http or https).
• higress.io/permanent-redirect-code: HTTP status code of permanent redirection, default is 301.

For example, to permanently redirect the request http://example.com/test to http://example.com/app, we can configure Higress through the following Annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/permanent-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /test
            pathType: Exact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

temporary redirect

• higress.io/temporal-redirect: The target URL of temporary redirection must contain scheme (http or https).

The request http://example.com/test needs to be temporarily redirected to http://example.com/app. We can configure Higress through the following Annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/temporal-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /test
            pathType: Exact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

Configure backend service protocol

Higress uses HTTP protocol by default to forward requests to the backend business container. When your business container uses the HTTPS protocol, you can use the annotation higress.io/backend-protocol: "HTTPS"; when your business container serves GRPC, you can use the annotation higress.io/backend-protocol: "GRPC ".

Compared with Nginx Ingress, if the Port Name in the K8s Service resource to which your backend service belongs is defined as grpc or http2, you do not need to configure the annotation higress.io/backend-protocol: "GRPC", Higress will automatically use GRPC. Or HTTP2.

For example, if we need to request example/test to be forwarded to the backend service using the HTTPS protocol, we can configure Higress through the following Annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/backend-protocol: "HTTPS"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /
            pathType: Exact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

If you need to forward the request example/test to the backend service using the GRPC protocol, the first way is to pass annotations, as shown below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/backend-protocol: "GRPC"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /test
            pathType: Exact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

The second method is to specify the Service Port Name as grpc, as shown below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /order
            pathType: Exact
---
apiVersion: v1
kind: Service
metadata:
  name: demo-service
spec:
  ports:
    - name: grpc # 指定 Service Port Name 为 grpc，Higress 会自动使用 GRPC 协议
      port: 80
      protocol: TCP
  selector:
    app: demo-service
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.

time out

Higress provides routing-level timeout settings. Unlike nginx ingress, there is no distinction between connection/read and write timeouts. Instead, the total delay for interface processing is configured. If not configured, there is no limit by default. For example, the backend does not return a response, and the gateway Will wait indefinitely.

Set the timeout to 5s:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/timeout: "5"
  name: demo
spec:
  ingressClassName: higress
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port:
                  number: 80
            path: /test
            pathType: Exact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

Grayscale release

Higress provides complex routing processing capabilities. Similar to Ingress Nginx, Higress also supports grayscale publishing functions based on Header, Cookie and weight. The grayscale publishing function can be implemented by setting annotations. In order to enable the grayscale publishing function, you need to set the annotation higress.io/canary: "true". Different grayscale publishing functions can be achieved through different annotations.

When multiple methods are configured at the same time, the grayscale method selection priority is: Based on Header > Based on Cookie > Based on weight (from high to low).

Below we use a sample application to illustrate the grayscale publishing function.

Step 1. Deploy the Production application

First create an application resource list for the production environment:

# production.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: production
  labels:
    app: production
spec:
  selector:
    matchLabels:
      app: production
  template:
    metadata:
      labels:
        app: production
    spec:
      containers:
        - name: production
          image: mirrorgooglecontainers/echoserver:1.10
          ports:
            - containerPort: 8080
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
---
apiVersion: v1
kind: Service
metadata:
  name: production
  labels:
    app: production
spec:
  ports:
    - port: 80
      targetPort: 8080
      name: http
  selector:
    app: production
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.

Then create an Ingress resource object for production environment access:

# production-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: production
spec:
  ingressClassName: higress
  rules:
    - host: echo.k8s.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: production
                port:
                  number: 80
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.

Directly create the above resource objects:

☸ ➜ kubectl apply -f production.yaml
☸ ➜ kubectl apply -f production-ingress.yaml
☸ ➜ kubectl get pods -l app=production
NAME                          READY   STATUS    RESTARTS   AGE
production-64cfc46b65-j6krb   1/1     Running   0          56s
☸ ➜ kubectl get ingress production
NAME         CLASS     HOSTS            ADDRESS   PORTS   AGE
production   higress   echo.k8s.local             80      9s
1.
2.
3.
4.
5.
6.
7.
8.

After the application is deployed successfully, you can access the application normally:

☸ ➜ curl http://echo.k8s.local


Hostname: production-64cfc46b65-j6krb

Pod Information:
        node name:      node1
        pod name:       production-64cfc46b65-j6krb
        pod namespace:  default
        pod IP: 10.0.1.249

Server values:
        server_versinotallow=nginx: 1.13.3 - lua: 10008

Request Information:
        client_address=10.0.1.162
        method=GET
        real path=/
        query=
        request_versinotallow=1.1
        request_scheme=http
        request_uri=http://echo.k8s.local:8080/

Request Headers:
        accept=*/*
        host=echo.k8s.local
        original-host=echo.k8s.local
        req-start-time=1713508680651
        user-agent=curl/7.87.0
        x-b3-sampled=0
        x-b3-spanid=5ec284bec822750c
        x-b3-traceid=5c48c3cc4192ddc85ec284bec822750c
        x-envoy-attempt-count=1
        x-envoy-decorator-operatinotallow=production.default.svc.cluster.local:80/*
        x-envoy-internal=true
        x-forwarded-for=192.168.0.112
        x-forwarded-proto=http
        x-request-id=d42c8455-5b2e-471d-811c-65f80210ccec

Request Body:
        -no body in request-
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.

Step 2. Create Canary version

Refer to the production.yaml file of the above Production version and create a Canary version of the application.

# canary.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: canary
  labels:
    app: canary
spec:
  selector:
    matchLabels:
      app: canary
  template:
    metadata:
      labels:
        app: canary
    spec:
      containers:
        - name: canary
          image: mirrorgooglecontainers/echoserver:1.10
          ports:
            - containerPort: 8080
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
---
apiVersion: v1
kind: Service
metadata:
  name: canary
  labels:
    app: canary
spec:
  ports:
    - port: 80
      targetPort: 8080
      name: http
  selector:
    app: canary
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.

Next, you can segment the traffic by configuring Annotation rules.

Release based on Header grayscale

When we need to perform grayscale publishing through Header, we can configure Higress's grayscale publishing strategy through the following Annotation:

• Only configure higress.io/canary-by-header: perform traffic segmentation based on the name of the Request Header. When the request contains this header and its value is always, the request traffic will be allocated to the grayscale service entrance; otherwise, the request traffic will not be allocated to the grayscale service.
• Configure higress.io/canary-by-header and higress.io/canary-by-header-value at the same time: traffic segmentation is performed based on the name and value of the Request Header. When the header name and header value in the request match this configuration, the request traffic will be allocated to the grayscale service; otherwise, the request traffic will not be allocated to the grayscale service.

For example, when the request header is higress: always, the grayscale service canary will be accessed; otherwise, the formal service production will be accessed, then we can create the following Ingress object:

# canary-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/canary: "true"
    higress.io/canary-by-header: "higress"
  name: canary
spec:
  ingressClassName: higress
  rules:
    - host: echo.k8s.local
      http:
        paths:
          - backend:
              service:
                name: canary
                port:
                  number: 80
            path: /
            pathType: Prefix
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.

Just apply the above YAML file directly:

☸ ➜ kubectl apply -f canary.yaml
☸ ➜ kubectl apply -f canary-ingress.yaml
☸ ➜ kubectl get pods -l app=canary
NAME                      READY   STATUS    RESTARTS   AGE
canary-7d97679b67-sh2r8   1/1     Running   0          12m
☸ ➜ kubectl get ingress canary
NAME     CLASS     HOSTS            ADDRESS   PORTS   AGE
canary   higress   echo.k8s.local             80      37s
1.
2.
3.
4.
5.
6.
7.
8.

After updating the above Ingress resource object, we add different Header values ​​to the request and visit the application domain name again to see the effect.

☸ ➜ for i in $(seq 1 10); do curl -s echo.k8s.local | grep "Hostname"; done
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

Here we did not configure any Header when making the request, so the request was not sent to the Canary application. What if it is set to other values:

☸ ➜ for i in $(seq 1 10); do curl -s -H "higress: always" echo.k8s.local | grep "Hostname"; done
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

Since the Header value set in our request is higress: always, all requests are routed to the canary service. If they are other values, they will not be routed to the canary service.

At this time, we can add a rule such as higress.io/canary-by-header-value: user-value based on the previous annotation (i.e. canary-by-header), so that requests with specified Header values ​​can be routed to The service specified in Canary Ingress.

annotations:
  higress.io/canary: "true" # 要开启灰度发布机制，首先需要启用 Canary
  higress.io/canary-by-header: "higress" # 基于header的流量切分
  higress.io/canary-by-header-value: "user-value"
1.
2.
3.
4.

After also updating the Ingress object, revisit the application. When the Request Header meets higress: user-value, all requests will be routed to the Canary version:

☸ ➜ for i in $(seq 1 10); do curl -s -H "higress: user-value" echo.k8s.local | grep "Hostname"; done
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

Grayscale publishing based on cookies

The usage rules of annotation based on Request Header are similar. For example, in an A/B testing scenario, users in Beijing need to be allowed to access the Canary version. Then when the cookie annotation is set to higress.io/canary-by-cookie: "users_from_Beijing", the background can check the logged in user request. If the user's access source is from Beijing, set the value of cookie users_from_Beijing to always, so This ensures that users in Beijing only access the Canary version.

Cookie-based grayscale publishing does not support customizing the value corresponding to Key, it can only be always.

For example, when the cookie we request now is users_from_Beijing=always, it will access the grayscale version service canary; in other cases, it will access the official service production, then we can create the following Ingress object:

# canary-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/canary: "true"
    higress.io/canary-by-cookie: "users_from_Beijing"
  name: canary
spec:
  ingressClassName: higress
  rules:
    - host: echo.k8s.local
      http:
        paths:
          - backend:
              service:
                name: canary
                port:
                  number: 80
            path: /
            pathType: Prefix
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.

After updating the above Ingress resource object, we set a cookie value of users_from_Beijing=always in the request and access the application domain name again.

☸ ➜ for i in $(seq 1 10); do curl -s -b "users_from_Beijing=always" echo.k8s.local | grep "Hostname"; done
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
Hostname: canary-7d97679b67-sh2r8
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

We can see that the applications are routed to the Canary version of the application. If we set this cookie value to another value, it will not be routed to the Canary application.

Release based on weighted grayscale

A typical application scenario of weight-based traffic segmentation is blue-green deployment, which can be achieved by setting the weight to 0 or 100. For example, you can set the Green version as the main part and configure the entrance of the Blue version as Canary. Initially, the weight is set to 0 so traffic is not proxied to the Blue version. Once the new version is successfully tested and verified, the weight of the Blue version can be set to 100, that is, all traffic will be transferred from the Green version to Blue.

In Higress, we can configure the weight-based grayscale publishing strategy through the following Annotation:

• higress.io/canary-weight: Set the percentage of requests to the specified service (the value is an integer from 0 to 100)
• higress.io/canary-weight-total: Set the total weight, the default is 100

For example, if we configure the weight of the grayscale service canary to be 20% and the weight of the formal service production to 80%, we can create the following Ingress object:

# canary-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/canary: "true"
    higress.io/canary-weight: "20" # 分配20%流量到当前Canary版本
  name: canary
spec:
  ingressClassName: higress
  rules:
    - host: echo.k8s.local
      http:
        paths:
          - backend:
              service:
                name: canary
                port:
                  number: 80
            path: /
            pathType: Prefix
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.

Update the above Ingress object, then we will continue to access this application in the command line terminal and observe the changes in Hostname:

☸ ➜ for i in $(seq 1 10); do curl -s echo.k8s.local | grep "Hostname"; done
Hostname: production-64cfc46b65-j6krb
Hostname: canary-7d97679b67-sh2r8
Hostname: production-64cfc46b65-j6krb
Hostname: canary-7d97679b67-sh2r8
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
Hostname: production-64cfc46b65-j6krb
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

Since we allocate about 20% of the traffic weight to the Canary version of the application, we visited the Canary version of the application 3 times out of 10 times (not necessarily), which is in line with our expectations.

At this point, we have implemented traffic segmentation through the grayscale publishing function of Higress. We can choose different grayscale publishing methods according to different scenarios.

plug-in

Higress provides three plug-in extension mechanisms: Wasm, Lua, and out-of-process. Customized function expansion can be achieved through the plug-in mechanism. Supports plug-ins written in multiple languages, and the effective granularity supports global level, domain name level, and routing level. Plug-ins support hot updates, and changes to plug-in logic and configuration will have no impact on traffic.

To enable plug-ins for Higress, there are two methods. One is to configure it through the Higress console, and the other is to configure it through the Higress WasmPlugin CRD.

Configure via Higress console

The Higress console provides 3 entrances for plug-in configuration:

• Global configuration: Plug-in Market -> Select plug-ins for configuration
• Domain name level configuration: Domain name management -> Select domain name -> Click policy -> Select plug-in for configuration
• Routing level configuration: Routing configuration->select routing->click policy->select plug-in for configuration

The effective priority of these three configurations is: routing level > domain name level > global , that is, the global configuration will only take effect for requests that do not match a specific route or domain name. 

For general plug-ins, including custom plug-ins, the routing/domain name level configuration fields and global configuration fields are exactly the same; for authentication plug-ins (Key authentication, HMAC authentication, Basic authentication, JWT authentication, etc.), it is different. The configuration only configures Consumer credentials and whether to enable global authentication, and configures the list of Consumers allowed to access through the allow field at the routing/domain name level.

For example, if we want to configure a Basic Auth plug-in for the previous foo.bar.com service, we first need to enable the Basic Auth plug-in in the global configuration, and then configure the Basic Auth plug-in at the domain name level (or routing level).

Switch to the plugin market in the console, find the Basic Auth plugin, click Configure, and then configure user credentials through fields in the data editor , as shown below: consumers 

global_auth: false
consumers: # 配置两个用户凭证
  - name: consumer1
    credential: admin:123456
  - name: consumer2
    credential: guest:abc
1.
2.
3.
4.
5.
6.

Remember to select the On status and click Save.

Configure Basic Authentication

Next, switch to the domain name management page, add a domain name foo.bar.com, then click Policy and select the Basic Auth plug-in to configure.

Configure the users allowed access through the allow field in the data editor, as follows:

allow:
  - consumer1
1.
2.

Remember to select the On status and click Save. Here we configure that only the consumer1 user is allowed to access the foo.bar.com service, and other callers are not allowed to access it.

Now we can access the foo.bar.com service to verify:

$ curl -v foo.bar.com/foo
*   Trying 192.168.0.116:80...
* Connected to foo.bar.com (192.168.0.116) port 80 (#0)
> GET /foo HTTP/1.1
> Host: foo.bar.com
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< www-authenticate: Basic realm=MSE Gateway
< date: Sat, 20 Apr 2024 06:05:15 GMT
< server: istio-envoy
< content-length: 0
<
* Connection #0 to host foo.bar.com left intact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.

You can see that 401 Unauthorized was returned because we did not provide user credentials. Next we provide user credentials for access:

$ curl -v -u admin:123456 foo.bar.com/foo
*   Trying 192.168.0.116:80...
* Connected to foo.bar.com (192.168.0.116) port 80 (#0)
* Server auth using Basic with user 'admin'
> GET /foo HTTP/1.1
> Host: foo.bar.com
> Authorization: Basic YWRtaW46MTIzNDU2
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-app-name: http-echo
< x-app-version: 0.2.4
< date: Sat, 20 Apr 2024 06:06:10 GMT
< content-length: 4
< content-type: text/plain; charset=utf-8
< req-cost-time: 1
< req-arrive-time: 1713593170671
< resp-start-time: 1713593170673
< x-envoy-upstream-service-time: 1
< server: istio-envoy
<
foo
* Connection #0 to host foo.bar.com left intact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.

You can see that we successfully accessed the service after providing the user credentials. If the user credentials we provided are not in the list of users allowed to access, a 403 Forbidden error will be returned.

$ curl -v -u guest:abc foo.bar.com/foo
*   Trying 192.168.0.116:80...
* Connected to foo.bar.com (192.168.0.116) port 80 (#0)
* Server auth using Basic with user 'guest'
> GET /foo HTTP/1.1
> Host: foo.bar.com
> Authorization: Basic Z3Vlc3Q6YWJj
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< www-authenticate: Basic realm=MSE Gateway
< date: Sat, 20 Apr 2024 06:06:54 GMT
< server: istio-envoy
< content-length: 0
<
* Connection #0 to host foo.bar.com left intact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.

Configure via Higress WasmPlugin

In addition to configuring plug-ins through the console, you can also configure them through Higress WasmPlugin CRD. Higress WasmPlugin CRD is an extension of Istio WasmPlugin CRD and adds some new configuration fields.

Configuration field description for each item in matchRules:

Similarly, if you want to configure and implement the Basic Auth plug-in through WasmPlugin, you first need to create a WasmPlugin CRD object, as shown below:

# basic-auth-plugin.yaml
apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  name: basic-auth
  namespace: higress-system
spec:
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/basic-auth:1.0.0 # 插件镜像地址
  defaultConfig:
    consumers:
      - name: consumer1
        credential: admin:123456
      - name: consumer2
        credential: guest:abc
  matchRules:
    - domain:
        - foo.bar.com
      config:
        allow:
          - consumer2
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.

Reset the Basic Auth plug-in configuration configured through the console above and disable the Basic Auth plug-in.

In the above WasmPlugin object, we configured the user credentials through the defaultConfig field. This part actually corresponds to the global configuration in the console, and then configured the domain name level configuration through the matchRules field. This part corresponds to the domain name level configuration in the console. Of course In addition to domain, you can also configure the ingress field, which corresponds to the routing-level configuration in the console. For example, the above configuration indicates that only consumer2 user is allowed to access the foo.bar.com service.

Create the WasmPlugin object above directly:

☸ ➜ kubectl apply -f basic-auth-plugin.yaml
1.

Then visit the foo.bar.com service again to verify:

☸ ➜ curl -v foo.bar.com/foo
*   Trying 192.168.0.116:80...
* Connected to foo.bar.com (192.168.0.116) port 80 (#0)
> GET /foo HTTP/1.1
> Host: foo.bar.com
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< www-authenticate: Basic realm=MSE Gateway
< date: Sat, 20 Apr 2024 06:23:48 GMT
< server: istio-envoy
< content-length: 0
<
* Connection #0 to host foo.bar.com left intact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.

You can see that 401 Unauthorized was returned because we did not provide user credentials. Next we provide user credentials for access:

☸ ➜ curl -v -u guest:abc foo.bar.com/foo
*   Trying 192.168.0.116:80...
* Connected to foo.bar.com (192.168.0.116) port 80 (#0)
* Server auth using Basic with user 'guest'
> GET /foo HTTP/1.1
> Host: foo.bar.com
> Authorization: Basic Z3Vlc3Q6YWJj
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-app-name: http-echo
< x-app-version: 0.2.4
< date: Sat, 20 Apr 2024 06:24:49 GMT
< content-length: 4
< content-type: text/plain; charset=utf-8
< req-cost-time: 1
< req-arrive-time: 1713594289845
< resp-start-time: 1713594289847
< x-envoy-upstream-service-time: 0
< server: istio-envoy
<
foo
* Connection #0 to host foo.bar.com left intact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.

Above, we configured in WasmPlugin that only consumer2 users are allowed to access the foo.bar.com service, so only requests that provide consumer2 user credentials can access the service. Therefore, the guest:abc user credentials we provided here successfully accessed the service. If the provided If the user's credentials are not in the list of users allowed to access, a 403 Forbidden error will be returned.

☸ ➜ curl -v -u admin:123456 foo.bar.com/foo
*   Trying 192.168.0.116:80...
* Connected to foo.bar.com (192.168.0.116) port 80 (#0)
* Server auth using Basic with user 'admin'
> GET /foo HTTP/1.1
> Host: foo.bar.com
> Authorization: Basic YWRtaW46MTIzNDU2
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< www-authenticate: Basic realm=MSE Gateway
< date: Sat, 20 Apr 2024 06:26:00 GMT
< server: istio-envoy
< content-length: 0
<
* Connection #0 to host foo.bar.com left intact
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.

Here we have implemented the configuration of the Basic Auth plug-in through the Higress WasmPlugin CRD object to implement the user credential authentication function. In addition to the Basic Auth plug-in, Higress also provides many other plug-ins, such as Key Auth plug-in, HMAC Auth plug-in, JWT Auth plug-in, etc. You can choose different plug-ins for configuration according to actual needs. The configuration properties of each plug-in can be referred to the plug-in in the console. Market allocation instructions.

Custom plugin

In addition to the built-in plug-ins provided by Higress, sometimes we may have some special needs. In this case, we can implement customized function extensions through custom plug-ins. We can use a variety of programming languages ​​to write plug-ins. The effective granularity supports global level, domain name level, and routing level. In addition, plug-ins also support hot updates. Changing plug-in logic and configuration will not damage the traffic. Below we use a simple example to illustrate how to write a custom plug-in.

First, you need to install Golang and TinyGo. Golang requires version 1.18 or above, the official guide link: https://go.dev/doc/install; TinyGo requires version 0.28.1 or above, the official guide link: https://tinygo.org/getting-started/install/.

☸ ➜ go version
go version go1.20.14 darwin/arm64
☸ ➜ tinygo version
tinygo version 0.30.0 darwin/amd64 (using go version go1.20.14 and LLVM version 16.0.1)
1.
2.
3.
4.

Next, we first initialize the project directory, create a new directory named, and execute the following command in the created directory to initialize the Go project. wasm-demo-go 

go mod init wasm-demo-go
1.

Domestic environment requires setting up an agent to download dependency packages

go env -w GOPROXY=https://proxy.golang.com.cn,direct
1.

Then download the dependencies for building the plugin:

go get github.com/higress-group/proxy-wasm-go-sdk
go get github.com/alibaba/higress/plugins/wasm-go@main
go get github.com/tidwall/gjson
1.
2.
3.

Next, let's write a simple plug-in to implement a simple function. When the plug-in configuration has mockEnable: true, it directly returns the hello world response; when the plug-in is not configured, or when mockEnable: false is set, the hello: world request is added to the original request. Header header.

package main

import (
    "github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
    "github.com/higress-group/proxy-wasm-go-sdk/proxywasm"
    "github.com/higress-group/proxy-wasm-go-sdk/proxywasm/types"
    "github.com/tidwall/gjson"
)

func main() {
    wrapper.SetCtx(
        // 插件名称
        "my-plugin",
        // 为解析插件配置，设置自定义函数
         wrapper.ParseConfigBy(parseConfig),
        // 为处理请求头，设置自定义函数
        wrapper.ProcessRequestHeadersBy(onHttpRequestHeaders),
    )
}

// 自定义插件配置
type MyConfig struct {
    mockEnable bool
}

// 在控制台插件配置中填写的yaml配置会自动转换为json，此处直接从json这个参数里解析配置即可
func parseConfig(json gjson.Result, config *MyConfig, log wrapper.Log) error {
    // 解析出配置，更新到config中
    config.mockEnable = json.Get("mockEnable").Bool()
    return nil
}

func onHttpRequestHeaders(ctx wrapper.HttpContext, config MyConfig, log wrapper.Log) types.Action {
    proxywasm.AddHttpRequestHeader("hello", "world")
    if config.mockEnable {
        proxywasm.SendHttpResponse(200, nil, []byte("hello world"), -1)
    }
    return types.ActionContinue
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.

The plug-in configuration in the gateway console is in yaml format, which will be automatically converted to json format when delivered to the plug-in, so the parseConfig in the example can directly parse the configuration from json.

In the above code, the custom function onHttpRequestHeadersBy is used to process requests in the HTTP request header processing stage through wrapper.ProcessRequestHeadersBy. In addition, you can also set custom processing functions for other stages in the following ways.

proxywasm.AddHttpRequestHeader and proxywasm.SendHttpResponse in the above example code are two tool methods provided by the plug-in SDK. The main tool methods are shown in the following table:

Next we need to compile and generate the WASM file:

go mod tidy
tinygo build -o main.wasm -scheduler=none -target=wasi -gc=custom -tags="custommalloc nottinygc_finalizer" ./main.go
1.
2.

Successful compilation will generate the file main.wasm in the current directory. To configure the plug-in interactively with the Wasmplugin CRD or Console UI in Higress, the wasm file needs to be packaged into an oci or docker image.

Create a Dockerfile file under the project root directory with the following content:

FROM scratch
COPY main.wasm plugin.wasm
1.
2.

Then execute the following commands to build and push the image:

docker build -t cnych/higress-plugin-demo:1.0.0 .
docker push cnych/higress-plugin-demo:1.0.0
1.
2.

In this way, we package the custom plug-in into a Docker image. Then create a custom plug-in in the plug-in market of the gateway console and fill in the Wasm image address built above.

add plugin

After creation, click the configuration button of the plug-in card, fill in the plug-in configuration mockEnable: true, and turn on the enable switch to take effect. If the plug-in logic changes, you can build a new image and use a different image tag. Click the edit button in the menu at the top right of the plug-in card and change the Wasm image address to the address of the new version.

However, it should be noted that after it is enabled, it will take effect on all requests, so it needs to be used with caution in a production environment. You can control the effective scope of the plug-in by configuring the plug-in at the domain name level or routing level.

In addition to configuring the plug-in through the console, you can also configure the plug-in through the Higress WasmPlugin CRD object, which allows you to more flexibly control the effective scope of the plug-in.

# wasm-plugin-demo.yaml
apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  name: plugin-demo
  namespace: higress-system
spec:
  url: oci://docker.io/cnych/higress-plugin-demo:1.0.0
  defaultConfig:
    mockEnable: false
  matchRules:
    - domain:
        - foo.bar.com
      config:
        mockEnable: true
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.

Apply the WasmPlugin object above directly:

☸ ➜ kubectl apply -f wasm-plugin-demo.yaml
1.

Then we access the foo.bar.com service again to verify:

☸ ➜ curl foo.bar.com/foo
hello world
1.
2.

You can see that after we provide the plug-in configuration mockEnable: true, the hello world response is returned directly. If we set the plug-in configuration to mockEnable: false, the hello: world request header will be added to the original request.

In this way, we configure the custom plug-in through the Higress WasmPlugin CRD object to implement custom function expansion.

Here is a brief explanation of the plug-in’s effectiveness mechanism:

• The user compiles the code into a wasm file
• The user builds the wasm file into a docker image
• The user pushes the docker image to the mirror warehouse
• User creates WasmPlugin resource
• Istio watch to WasmPlugin resource changes
• The xDS proxy process in Higress Gateway obtains the configuration from Istio and discovers the mirror address of the plug-in.
• xDS proxy pulls the image from the image warehouse
• xDS proxy extracts the wasm file from the image
• The envoy process in Higress Gateway obtains the configuration from xDS proxy and discovers the local path of the wasm file
• Envoy loads wasm files from local files

Plug-in takes effect

Here envoy obtains the configuration and loads the wasm file using the ECDS (Extension Config Discovery Service) mechanism, which implements the wasm file update and direct hot loading without causing any connection interruption, and the business traffic is completely intact.

In addition to the Higress functions introduced in this article, there are many other functions and best practices. For more information, please refer to the Higress official documentation.

Reference documentation: https://higress.io