Having applied that configuration, we would then dump MachineConfig objects applying to worker nodes:<\/p>\n
\nDUMP=$(oc get machineconfig | grep -v rendered | \\
\n awk ‘\/worker\/{print $1}’ | tr ‘\\n’ ‘ ‘)<\/p>\n
oc get machineconfig -o yaml $DUMP >machineconfig-infra.yaml\n<\/p>\n<\/blockquote>\n
We would then edit machineconfig-infra.yaml<\/i> content, removing “generated-by” annotations, creationTimestamps, generation, ownerReferences, resourceVersions, selfLink and uid metadata. Replace any remaning mention of “worker” by “infra”. Then apply the resulting objects:<\/p>\n\noc apply -f machineconfig-infra.yaml
\noc get mc
\n00-infra 2.2.0 1m
\n01-infra-container-runtime 2.2.0 1m
\n01-infra-kubelet 2.2.0 1m
\n99-infra-ad9f8790-f270-11e9-a34e-525400e1605b-registries 2.2.0 1m
\n99-infra-ssh 2.2.0 1m\n<\/p>\n<\/blockquote>\n
At that stage, the MachineConfig Operator should be rendering a last MachineConfig object, including an exhaustive list of configurations for our infra nodes. Once oc get mc<\/i> includes that rendered configuration, we would make sure the MachineConfig Operator is done with our MachineConfigPool and start re-labeling nodes accordingly:<\/p>\n\noc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED
infra rendered-infra-0506920a222781a19fff88a4196deef4 True False False
master rendered-master-747943425e64364488e51d15e5281265 True False False
worker rendered-worker-5e70256103cc4d0ce0162430de7233a1 True False False
oc label node infra1.nodes.example.com node-role.kubernetes.io\/infra=
\nnode\/infra1.nodes.example.com labeled
oc label node infra1.nodes.example.com node-role.kubernetes.io\/worker-
\nnode\/infra1.nodes.example.com labeled<\/p>\n<\/blockquote>\n
From there, our node would be set unschedulable, drained, and rebooted. Our customized MachineConfig should have changed the role label applied when our node boots, which we may confirm once it is done restarting<\/p>\n
\noc get nodes
\ncompute1.nodes.example.com Ready worker 47m v1.14.6+c07e432da
\ncompute2.nodes.example.com Ready worker 45m v1.14.6+c07e432da
\ncompute3.nodes.example.com Ready worker 34m v1.14.6+c07e432da
\ncompute4.nodes.example.com Ready worker 33m v1.14.6+c07e432da
\ncompute5.nodes.example.com Ready worker 31m v1.14.6+c07e432da
\ninfra1.nodes.example.com Ready infra 2h v1.14.6+c07e432da
\ninfra2.nodes.example.com Ready worker 2h v1.14.6+c07e432da
\ninfra3.nodes.example.com Ready worker 2h v1.14.6+c07e432da
\nmaster1.nodes.example.com Ready master 2h v1.14.6+c07e432da
\nmaster2.nodes.example.com Ready master 2h v1.14.6+c07e432da
\nmaster3.nodes.example.com Ready master 2h v1.14.6+c07e432da\n<\/p>\n<\/blockquote>\n
Once our node is back, we would proceed with the next infra node.<\/p>\n
We would eventually reconfigure our Ingress Controller deploying OpenShift Routers back to our infra nodes:<\/p>\n
\noc edit -n openshift-ingress-operator ingresscontroller default
\nspec:
nodePlacement:
nodeSelector:
matchLabels:
node-role.kubernetes.io\/infra: “”
replicas: 3<\/p>\n<\/blockquote>\n
We would then keep track of routers Pods as they’re being re-deployed:<\/p>\n
\n\noc get pods -n openshift-ingress -o wide
\nNAME READY STATUS RESTARTS AGE IP NODE
router-default-86cdb97784-4d72k 1\/1 Running 0 14m 10.42.253.14 infra2.nodes.example.com
\nrouter-default-86cdb97784-8f5vm 1\/1 Running 0 14m 10.42.253.15 infra3.nodes.example.com
router-default-86cdb97784-bvvdc 1\/1 Running 0 105s 10.42.253.13 infra1.nodes.example.com<\/p>\n<\/blockquote>\n\n\n
<\/p>\n\n\n
Ceph RBD Storage<\/strong><\/em><\/h2>\nLater on, we may want to configure OpenShift interfacing with an existing Ceph cluster, setting up persisting volumes.<\/p>\n
While OpenShift 3 used to ship with rbd<\/i> binaries in the api controller image, while allowing for their installation on OpenShift nodes, this is no longer the case with OpenShift 4. Instead, we would rely on CSI (Container Storage Interface), which is meant to be a more generic interface.<\/p>\n
Then, we would need to deploy Ceph CSI interface to OpenShift,<\/p>\n
\ngit clone https:\/\/github.com\/ceph\/ceph-csi\/
oc new-project ceph-csi
for sa in rbd-csi-provisioner rbd-csi-nodeplugin; do
\n oc create sa $sa
oc adm policy add-scc-to-user hostaccess system:serviceaccount:ceph-csi:$sa
oc adm policy add-scc-to-user privileged system:serviceaccount:ceph-csi:$sa
done
\ncat ceph-csi\/deploy\/rbd\/kubernetes\/v1.14+\/csi-*yaml | sed ‘s|namespace: default|namespace: ceph-csi|g’ | oc apply -n ceph-csi -f-
cat <<EOF >config.json
[
{
“clusterID”: “my-ceph-cluster-id”,
“monitors: [ “10.1.2.3”,”10.1.2.4″10.1.2.5″ ]
}
]
EOF
oc delete cm -n ceph-csi ceph-csi-config
oc create cm -n ceph-csi ceph-csi-config –from-file=config.json=.\/config.json
cat << EOF >secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-rbd-secret
stringData:
userID: my-ceph-user-id
userKey: my-user-key
EOF
oc apply -n default -f secret.yaml
cat << EOF >storageclass.yaml
apiVersion: storage.k8s.io\/v1
kind: StorageClass
metadata:
name: ceph-storage
provisioner: rbd.csi.ceph.com
parameters:
clusterID: my-ceph-cluster-id
pool: kube
imageFeatures: layering
csi.storage.k8s.io\/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io\/provisioner-secret-namespace: default
csi.storage.k8s.io\/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io\/node-stage-secret-namespace: default
csi.storage.k8s.io\/fstype: xfs
reclaimPolicy: Delete
mountOptions:
– discard
EOF
oc apply -f storageclass.yaml<\/p>\n<\/blockquote>\n
At that stage, we would have deployed a DaemonSet of csi-rbdplugin<\/i> Pods, tasked with attaching and detaching volumes during Pods scheduling and terminations, as well as a Deployment of csi-rbdplugin-provisioner<\/i> Pods, creating and purging volumes out of Ceph, while managing OpenShift Persistent Volumes.<\/p>\n
At that stage, we may create a first Persistent Volume and redeploy OpenShift integrated registry on top of it:<\/p>\n
\ncat <<EOF >registry-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: image-registry
namespace: openshift-image-registry
spec:
accessModes:
– ReadWriteOnce
resources:
requests:
storage: 100Gi
EOF
oc apply -f registry-pvc.yaml
oc edit configs.imageregistry.operator.openshift.io
[…]
storage:
pvc:
claim: image-registry-storage
[…]
oc get pods -n openshift-image-registry -w<\/p>\n<\/blockquote>\n\n\n
<\/p>\n\n\n
Conclusion<\/strong><\/em><\/h2>\nFirst thing I would regret is the disappearance of rbd binaries from controllers images. As a result, the Ceph provisioner we used to configure with OpenShift3 no longer works. Apparently, CSI provisioners would be recommended instead, though that implementation is kind of slower, and involves quite a lot of Pods.<\/p>\n
After deployment, roughly 12G RAM and 4 CPUs are allocated to cluster operators and OpenShift internals.<\/p>\n
Another concern may be that all those operators are privileged actors in our cluster. While we usually had to compromise a node to attack a cluster, now we have a lot of operators that might be accessed through the API, arguably expanding OpenShift attack surface.<\/p>\n
The dashboard shows a total CPU capacity of “100%”, which is quite useless.<\/p>\n
OpenShift 4.2 is based on Kubernetes 1.14. Among nother novelties, as compared with OpenShift 3, we could mention Istio reaching GA or Tekton pipelines.<\/p>","protected":false},"excerpt":{"rendered":"
Once again, quick post regarding OpenShift, today experimenting with the new installer, and OpenShift 4. First, let’s remind ourselves that OKD 4 has not yet been released. I would be using my RedHat account credentials pulling images. I usually refuse to touch anything that is not strictly open source (and freely distributed), though I would […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[12,13,2],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/860"}],"collection":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=860"}],"version-history":[{"count":10,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/860\/revisions"}],"predecessor-version":[{"id":875,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/860\/revisions\/875"}],"wp:attachment":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=860"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=860"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=860"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"OpenShift4](\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2019\/10\/dashboard-ocp4-1024x522.png\")
<\/a>