K8s : Bringing load-balancing to multus workloads with loxilb

In Kubernetes world, multus plugin has been gaining prominence as a way to introduce secondary networks to Pods. There are many use-cases which benefit from this :

• Attaching a SRIOV interface to a POD for high-performance apps
• Exploit multi-homing features with multiple interfaces

It gives great flexibility to the users but at the same time has certain downsides :

• No load-balancer services are available on these secondary networks
• It bypasses built-in k8s access-control and security policies

This blog post will discuss how to bring load-balancer services in a seamless manner to multus based workloads. In one of my last posts, I explored loxilb which has been slowly gaining momentum as an external LB in the cloud-native world. Support for seamless LB services on multus workloads will really help multus users scale their workloads and also ease the enforcement of policy/access control (PS — This loxilb feature might be in beta). Some folks have proposed exotic schemes (for exposing multus services) but a common complaint about such schemes is they break the tried and tested conventions of Kubernetes.

We will use k3s as the Kubernetes distribution. The step to install k3s is already covered in my previous blog. The major difference in k3s installation with multus (as compared to previous blog) is to use a slightly older version of k3s (v1.22.9+k3s1) due to certain compatibility issues with multus.

$ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.22.9+k3s1 INSTALL_K3S_EXEC="server --disable traefik --disable servicelb --disable-cloud-controller --kubelet-arg cloud-provider=external" K3S_KUBECONFIG_MODE="644" sh -

Overall, the idea is to create and test a topology similar to what is shown below :

Kubernetes topology with loxilb and multus

loxilb usually runs in an external node but several people have successfully run it as an inclusive Kubernetes component (daemon-set). For load-balancing multus pods, the external mode works best and hence we will use that. To install loxilb, we do the following:

1. Run docker in the external node following the guide here
2. Run kube-loxilb component of loxilb in Kubernetes following the guide here
3. Make sure the network connectivity is proper. When working with multus, we need to pay extra attention to IP address allocation and also need to make sure we have connectivity from loxilb node to Kubernetes worker node’s secondary interfaces used by multus (ensp0 and ens33 above). It is best to have them in a single LAN network.

Then, we install multus plugin. The steps are as follows:

1. Deploy multus daemonset using the instructions here
2. Create a NetworkAttachmentDefinition

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan1
spec:
  config: '{
            "cniVersion": "0.3.1",
            "type": "macvlan",
            "master": "ensp0",
            "mode": "bridge",
            "ipam": {
                "type": "host-local",
                "ranges": [
                    [ {
                         "subnet": "192.168.20.0/24",
                         "rangeStart": "192.168.20.20",
                         "rangeEnd": "192.168.20.200",
                         "gateway": "192.168.20.1"
                    } ]
                ]
            }
        }'
EOF

3. Create a test POD (with secondary attachment). Please note the annotations use the “networkattachment” name from previous step

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: pod-01
  labels:
    app: pod-01
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan1
spec:
  containers:
    - name: nginx-multus-test
      image: nginx:stable
      ports:
        - containerPort: 80
EOF

4. Create a test service for multus pod. The important thing here is the special loxilb annotation “loxilb.io/multus-nets” which denotes which multus networks should be considered while creating a LB rule. Once we have this annotation, loxilb will only create load-balancer entry to include secondary interface(s).

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: multus-service
  annotations:
    loxilb.io/multus-nets : macvlan1,macvlan2
spec:
  loadBalancerClass: loxilb.io/loxilb
  selector:
    app: pod-01
  ports:
    - port: 55003
      targetPort: 80
  type: LoadBalancer
EOF

Verifying everything :

1. Check pods are created properly

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   local-path-provisioner-84bb864455-qr2l5   1/1     Running   0          50m
kube-system   coredns-7796b77cd4-nd2p2                  1/1     Running   0          50m
kube-system   metrics-server-ff9dbcb6c-8sgvd            1/1     Running   0          50m
kube-system   kube-loxilb-f9c5cd878-25ppx               1/1     Running   0          49m
kube-system   kube-multus-ds-amd64-7kbms                1/1     Running   0          20m
default       pod-01                                    1/1     Running   0          6m41s

By default, macvlan plugin is not installed in k3s, so one might encounter an error in creating a pod :

$ kubectl get pods -A | grep pod-01
default       pod-01                               0/1     ContainerCreating   0          66s

$ kubectl describe pods pod-01 | grep -i "failed"
  Warning  FailedCreatePodSandBox  3s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e3e214462723c973ff5a343a9cff1e370fed5c4ccd570c8d84a408ede6419acb": plugin type="multus" name="multus-cni-network" failed (add): [default/pod-case-01:macvlan-conf-1]: error adding container to network "macvlan-conf-1": failed to find plugin "macvlan" in path [/opt/cni/bin /var/lib/rancher/k3s/data/995f5a281daabc1838b33f2346f7c4976b95f449c703b6f1f55b981966eba456/bin]

If we encounter this issue, we need to build and install macvlan plugin manually :

$ git clone https://github.com/containernetworking/plugins.git
$ cd plugins
$ ./build_linux.sh
$ ls bin/macvlan
$ sudo cp -f ./bin/macvlan /var/lib/rancher/k3s/data/current/bin/

2. Check services are created properly

$ kubectl get svc
NAME             TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)           AGE
kubernetes       ClusterIP      10.43.0.1       <none>          443/TCP           48m
nginx-service    ClusterIP      10.43.177.140   <none>          8080/TCP          47m
multus-service   LoadBalancer   10.43.134.27    123.123.123.1   55003:31250/TCP   4s

3. Check loxilb rules are created properly (inside node running loxilb pods)

$ docker exec -it loxilb loxicmd get lb -o wide
|  EXTERNAL IP  | PORT  | PROTOCOL | BLOCK | SELECT |  MODE   |  ENDPOINT IP   | TARGET PORT | WEIGHT |  STATE   |
|---------------|-------|----------|-------|--------|---------|----------------|-------------|--------|----------|
| 123.123.123.1 | 55003 | tcp      |     0 | rr     | default | 192.168.20.20  |          80 |     10 | -        |

4. Finally, send some traffic to validate the test topology

$ curl -s --connect-timeout 10 http://123.123.123.1:55003

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Conclusion — This feature of loxilb provides benefits of multus secondary interfaces together with well-known, tried and tested Kubernetes abstractions like LBs/Services etc.

(Disclaimer: The expressed opinions/views are purely personal)