If you’re not yet familiar with it, OpenShift is a container orchestration solution based on Kubernetes. Among others, it integrates with several storage providers such as Ceph.<\/p>\n\n\n\n
Although GlusterFS is probably the best choice in terms of OpenShift integration, we could argue Ceph is a better pick overall. And while this post doesn’t aim at offering an exhaustive comparison between the two, we could mention GlusterFS split-brains requiring manual recoveries, poor block devices performances, poor performances dealing with lots (100s) of volumes, the lack of kernel-land client dealing with file volumes, …<\/p>\n\n\n\n
<\/p>\n\n\n\n
The most common way to integrate Ceph with OpenShift is to register a StorageClass, as we could find in OpenShift documentations, managing Rados Block Devices.<\/p>\n\n\n\n
apiVersion: storage.k8s.io\/v1
kind: StorageClass
metadata:
annotations:
storageclass.beta.kubernetes.io\/is-default-class: “true”
name: ceph-storage
parameters:
adminId: kube
adminSecretName: ceph-secret-kube
adminSecretNamespace: default
monitors: 10.42.253.110:6789,10.42.253.111:6789,10.42.253.112:6789
pool: kube
userId: kube
userSecretName: ceph-secret-kube
userSecretNamespace: default
provisioner: kubernetes.io\/rbd
reclaimPolicy: Retain<\/p><\/blockquote>\n\n\n\nWe would also need to create a Secret, holding our Ceph client key. First, we would create our client, granting it with proper permissions:<\/p>\n\n\n\n
$> ceph auth get-or-create client.kube mon ‘allow r’ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=kube’ -o ceph.client.kube.keyring<\/p><\/blockquote>\n\n\n\n
Next, we would base64-encode our key:<\/p>\n\n\n\n
$> awk ‘\/^[ \\t]*key\/{print $3} ceph.client.kube.keyring | base64<\/p><\/blockquote>\n\n\n\n
And register our Secret, including our encoded secret:<\/p>\n\n\n\n
cat <<EOF | oc apply -n default -f-<\/p>
apiVersion: v1
data:
key: <base64-encoded-string>
kind: Secret
metadata:
name: ceph-secret-kube
type: kubernetes.io\/rbd<\/p>EOF<\/p><\/blockquote>\n\n\n\n
The previous configurations would then allow us to dynamically provision block devices deploying new applications to OpenShift.<\/p>\n\n\n\nAnd while block devices is a nice thing to have, dealing with stateful workloads such as databases, up until now, GlusterFS main advantage over Ceph was its ability to provide with ReadWriteMany volumes – that can be mounted from several Pods at once, as opposed to ReadWriteOnce or ReadWriteOnly volumes, that may only be accessed by one deployment, unless mounted as without write capabilities.<\/p>\n\n\n\n
<\/p>\n\n\n\n
On the other hand, in addition to Rados Block Devices, Ceph offers with an optional CephFS share, that is similar to NFS or GlusterFS, in that several clients can concurrently write the same folder. And while CephFS isn’t much mentioned reading through OpenShift documentations, Kubernetes officially supports it. Today, we would try and guess how to make that work with OpenShift.
CephFS is considered to be stable since Ceph 12 (Luminous), released a couple years ago. Since then, I’ve been working for a practical use case. Here it is.<\/p>\n\n\n\n