Tekton Integration Testing & Kubernetes Operators
Today, I tried to implement some integration tests for a Kubernetes controller, in the context of Tekton Pipelines.
Docker-in-Docker
I would run my tests on my own production cluster. I do not want to impact existing operations. As such, I want to run my tests in some isolated environment.
The Tekton Catalog gives a sample building a Docker container image, using a Docker-in-Docker sidecar container, offering with some Docker runtime.
This is typically used on Kubernetes clusters that don’t rely on the Docker container runtime (eg: Cri-O), or whenever we do not want to share the Kubernetes node’s Docker socket file to its containers – which is good security practice.
In our case, we could re-use such a sidecar, executing arbitrary containers, which would help running our tests isolated from the underlying Kubernetes cluster.
Kubernetes-in-Docker
“Kubernetes-in-Docker”, or “KIND”, is part of the Kubernetes SIGs project. It allows to easily deploy a Kubernetes cluster on top of Docker. While you would not use this deploying a production cluster, it’s a perfect solution running some tests.
Cluster topology can be customized. Runtime version can be chosen. Making this ideal running integration tests of Kubernetes controllers
Tekton
All we need is to write some Task, that would integrate Kubernetes-in-Docker, Docker-in-Docker, with the deployment and tests of your controller.
Here is one way to do it:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: kind-test-operator
spec:
params:
- default: docker.io/curlimages/curl:7.72.0
description: Image providing curl, pulling binaries required by this Task
name: init_image
type: string
- default: 1.21.1
description: Kubernetes cluster version
name: k8s_version
type: string
- default: 0.11.1
description: KIND version
name: kind_version
type: string
- default: docker.io/kindest/node
description: KinD Node Image repository, including Kubernetes in Docker runtime
name: kind_node_image
type: string
- default: docker.io/library/docker:stable
description: The location of the docker builder image.
name: builderimage
type: string
- default: docker dind
description: The location of the docker-in-docker image.
name: dindimage
type: string
steps:
# first, download kubectl client and kind binary
# next containers won't have curl
- args:
- -c
- |
set -x;
# install kubectl
curl -o /ci-bin/kubectl -fsL \
https://dl.k8s.io/release/v$(params.k8s_version)/bin/linux/amd64/kubectl;
chmod +x /ci-bin/kubectl;
# install kind
curl -o /ci-bin/kind -fsL \
https://github.com/kubernetes-sigs/kind/releases/download/v$(params.kind_version)/kind-linux-amd64;
chmod +x /ci-bin/kind;
test -x /ci-bin/kind -a -x /ci-bin/kubectl;
exit $?;
command:
- /bin/sh
image: $(params.init_image)
name: setup
securityContext:
runAsUser: 1000
volumeMounts:
- mountPath: /ci-bin
name: temp-bin
# next, using the Docker Builder Image, connecting to the Docker-in-Docker sidecar
# create a Kubernetes cluster, using kind
# deploy your operator, using kubectl
# and proceed with testing your controller
- args:
- -c
- |
export PATH=/ci-bin:$PATH;
# start kube cluster
kind create cluster --image=$(params.kind_node_image):v$(params.k8s_version);
# test cluster OK
kubectl get nodes
if ! kubectl get nodes 2>&1 | grep Ready >/dev/null; then
echo K8S KO - bailing out;
exit 1;
fi;
# deploy controller / adapt to fit your own usecase
kubectl create ns opsperator;
kubectl create -f $(workspaces.source.path)/deploy/kubernetes/crd.yaml;
kubectl create -f $(workspaces.source.path)/deploy/kubernetes/rbac.yaml;
kubectl create -f $(workspaces.source.path)/deploy/kubernetes/namespace.yaml;
grep -vE ' (resources|limits|memory|cpu|nodeSelector|node-role.kubernetes.io/.*):( |$)' \
$(workspaces.source.path)/deploy/kubernetes/run-ephemeral.yaml | kubectl apply -f-;
echo Waiting for operator to start ...;
while true;
do
kubectl get pods -n opsperator;
kubectl get pods -n opsperator | grep 1/1 >/dev/null && break;
sleep 10;
done;
# dummy test for controller
echo Creating test resource ...;
kubectl create ns collab-demo;
sed -e 's|do_network_policy.*|do_network_policy: false|' \
-e 's|do_exporters.*|do_exporters: false|' \
$(workspaces.source.path)/deploy/kubernetes/cr/draw.yaml \
| kubectl apply -f-;
echo Waiting for draw to start ...;
while true;
do
kubectl get draw -n collab-demo;
kubectl get draw -n collab-demo -o yaml | grep -A20 'status:' \
| grep 'ready: true' >/dev/null && break;
sleep 10;
done;
# check assets created by controller
echo Checking pods:;
kubectl get pods -n collab-demo -o wide;
echo Checking ingress:;
kubectl get ingress,svc -n collab-demo;
# if needed: include some additional steps, with proper runtime, testing your componets
echo Done;
exit 0;
command:
- /bin/sh
env:
- name: DOCKER_HOST
value: tcp://localhost:2376
- name: DOCKER_TLS_VERIFY
value: '1'
- name: DOCKER_CERT_PATH
value: /certs/client
image: $(params.builderimage)
name: kind
securityContext:
runAsUser: 1000
volumeMounts:
- mountPath: /ci-bin
name: temp-bin
- mountPath: /certs/client
name: dind-certs
# the Docker-in-Docker Sidecar Container
# where your Kubernetes-in-Docker cluster is being executed
sidecars:
- args:
- --storage-driver=vfs
- --userland-proxy=false
env:
- name: DOCKER_TLS_CERTDIR
value: /certs
image: $(params.dindimage)
name: dind
readinessProbe:
periodSeconds: 1
exec:
command:
- ls
- /certs/client/ca.pem
securityContext:
privileged: true
volumeMounts:
- mountPath: /certs/client
name: dind-certs
volumes:
- name: temp-bin
emptyDir: {}
- name: dind-certs
emptyDir: {}
workspaces:
- name: source
The steps deploying your controller and testing its functioning properly would vary. The example above includes some hardcoded commands for simplicity. Scaling out, you may want to figure out some generic way of proceeding — repositories that would respect some naming convention providing with sample deployment configurations, and unit testing scripts.
Conclusion
This may not be the best way to proceed. If you can afford to run your tests on some actual cluster, without affecting its operations, then this would be easier. You may query the Kubernetes cluster API hosting your Tekton installation, rather than bootstraping Kubernetes in Kubernetes.
Still this was fun to look at. Kubernetes running in Docker-in-Docker. in a Kubernetes. That doesn’t use Docker.