K8s : Bringing load-balancing to multus workloads with loxilb
In Kubernetes world, multus plugin has been gaining prominence as a way to introduce secondary networks to Pods. There are many use-cases which benefit from this :
- Attaching a SRIOV interface to a POD for high-performance apps
- Exploit multi-homing features with multiple interfaces
It gives great flexibility to the users but at the same time has certain downsides :
- No load-balancer services are available on these secondary networks
- It bypasses built-in k8s access-control and security policies
This blog post will discuss how to bring load-balancer services in a seamless manner to multus based workloads. In one of my last posts, I explored loxilb which has been slowly gaining momentum as an external LB in the cloud-native world. Support for seamless LB services on multus workloads will really help multus users scale their workloads and also ease the enforcement of policy/access control (PS — This loxilb feature might be in beta). Some folks have proposed exotic schemes (for exposing multus services) but a common complaint about such schemes is they break the tried and tested conventions of Kubernetes.
We will use k3s as the Kubernetes distribution. The step to install k3s is already covered in my previous blog. The major difference in k3s installation with multus (as compared to previous blog) is to use a slightly older version of k3s (v1.22.9+k3s1) due to certain compatibility issues with multus.
$ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.22.9+k3s1 INSTALL_K3S_EXEC="server --disable traefik --disable servicelb --disable-cloud-controller --kubelet-arg cloud-provider=external" K3S_KUBECONFIG_MODE="644" sh -
Overall, the idea is to create and test a topology similar to what is shown below :
loxilb usually runs in an external node but several people have successfully run it as an inclusive Kubernetes component (daemon-set). For load-balancing multus pods, the external mode works best and hence we will use that. To install loxilb, we do the following:
- Run docker in the external node following the guide here
- Run kube-loxilb component of loxilb in Kubernetes following the guide here
- Make sure the network connectivity is proper. When working with multus, we need to pay extra attention to IP address allocation and also need to make sure we have connectivity from loxilb node to Kubernetes worker node’s secondary interfaces used by multus (ensp0 and ens33 above). It is best to have them in a single LAN network.
Then, we install multus plugin. The steps are as follows:
- Deploy multus daemonset using the instructions here
- Create a NetworkAttachmentDefinition
cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan1
spec:
config: '{
"cniVersion": "0.3.1",
"type": "macvlan",
"master": "ensp0",
"mode": "bridge",
"ipam": {
"type": "host-local",
"ranges": [
[ {
"subnet": "192.168.20.0/24",
"rangeStart": "192.168.20.20",
"rangeEnd": "192.168.20.200",
"gateway": "192.168.20.1"
} ]
]
}
}'
EOF
3. Create a test POD (with secondary attachment). Please note the annotations use the “networkattachment” name from previous step
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: pod-01
labels:
app: pod-01
annotations:
k8s.v1.cni.cncf.io/networks: macvlan1
spec:
containers:
- name: nginx-multus-test
image: nginx:stable
ports:
- containerPort: 80
EOF
4. Create a test service for multus pod. The important thing here is the special loxilb annotation “loxilb.io/multus-nets” which denotes which multus networks should be considered while creating a LB rule. Once we have this annotation, loxilb will only create load-balancer entry to include secondary interface(s).
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
name: multus-service
annotations:
loxilb.io/multus-nets : macvlan1,macvlan2
spec:
loadBalancerClass: loxilb.io/loxilb
selector:
app: pod-01
ports:
- port: 55003
targetPort: 80
type: LoadBalancer
EOF
Verifying everything :
- Check pods are created properly
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-84bb864455-qr2l5 1/1 Running 0 50m
kube-system coredns-7796b77cd4-nd2p2 1/1 Running 0 50m
kube-system metrics-server-ff9dbcb6c-8sgvd 1/1 Running 0 50m
kube-system kube-loxilb-f9c5cd878-25ppx 1/1 Running 0 49m
kube-system kube-multus-ds-amd64-7kbms 1/1 Running 0 20m
default pod-01 1/1 Running 0 6m41s
By default, macvlan plugin is not installed in k3s, so one might encounter an error in creating a pod :
$ kubectl get pods -A | grep pod-01
default pod-01 0/1 ContainerCreating 0 66s
$ kubectl describe pods pod-01 | grep -i "failed"
Warning FailedCreatePodSandBox 3s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e3e214462723c973ff5a343a9cff1e370fed5c4ccd570c8d84a408ede6419acb": plugin type="multus" name="multus-cni-network" failed (add): [default/pod-case-01:macvlan-conf-1]: error adding container to network "macvlan-conf-1": failed to find plugin "macvlan" in path [/opt/cni/bin /var/lib/rancher/k3s/data/995f5a281daabc1838b33f2346f7c4976b95f449c703b6f1f55b981966eba456/bin]
If we encounter this issue, we need to build and install macvlan plugin manually :
$ git clone https://github.com/containernetworking/plugins.git
$ cd plugins
$ ./build_linux.sh
$ ls bin/macvlan
$ sudo cp -f ./bin/macvlan /var/lib/rancher/k3s/data/current/bin/
2. Check services are created properly
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 48m
nginx-service ClusterIP 10.43.177.140 <none> 8080/TCP 47m
multus-service LoadBalancer 10.43.134.27 123.123.123.1 55003:31250/TCP 4s
3. Check loxilb rules are created properly (inside node running loxilb pods)
$ docker exec -it loxilb loxicmd get lb -o wide
| EXTERNAL IP | PORT | PROTOCOL | BLOCK | SELECT | MODE | ENDPOINT IP | TARGET PORT | WEIGHT | STATE |
|---------------|-------|----------|-------|--------|---------|----------------|-------------|--------|----------|
| 123.123.123.1 | 55003 | tcp | 0 | rr | default | 192.168.20.20 | 80 | 10 | - |
4. Finally, send some traffic to validate the test topology
$ curl -s --connect-timeout 10 http://123.123.123.1:55003
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Conclusion — This feature of loxilb provides benefits of multus secondary interfaces together with well-known, tried and tested Kubernetes abstractions like LBs/Services etc.
(Disclaimer: The expressed opinions/views are purely personal)