{"id":764,"date":"2018-06-09T23:35:23","date_gmt":"2018-06-09T21:35:23","guid":{"rendered":"https:\/\/blog.unetresgrossebite.com\/?p=764"},"modified":"2018-09-02T19:09:29","modified_gmt":"2018-09-02T17:09:29","slug":"openshift-supervision","status":"publish","type":"post","link":"https:\/\/blog.unetresgrossebite.com\/?p=764","title":{"rendered":"OpenShift Supervision"},"content":{"rendered":"

Today I am looking back on a few topics I had a hard time properly deploying using OpenShift 3.7 and missing proper dynamic provisioning despite a poorly-configured GlusterFS cluster.
\nSince then, I deployed a 3 nodes Ceph cluster, using Sebastien Han’s ceph-ansible playbooks, allowing me to further experiment with persistent volumes.
\nAnd OpenShift Origin 3.9 also came out, shipping with various fixes and new features, such Gluster Block volumes support, that might address some of GlusterFS performances issues.<\/p>\n

 <\/p>\n

OpenShift Ansible playbooks include a set of roles focused on collecting and making sense out of your cluster metrics, starting with Hawkular.<\/p>\n

We could set up a few Pods running Hawkular, Heapster to collect data from your OpenShift nodes and a Cassandra database to store them, defining the following variables and applying the playbooks\/openshift-metrics\/config.yml<\/em> playbook:<\/p>\n

\"Hawkular\"<\/a>

Hawkular integration with OpenShift<\/p><\/div>\n

openshift_metrics_cassandra_limit_cpu: 3000m<\/em>
\nopenshift_metrics_cassandra_limit_memory: 3Gi<\/em>
\nopenshift_metrics_cassandra_node_selector: {“region”:”infra”}<\/em>
\nopenshift_metrics_cassandra_pvc_prefix: hawkular-metrics<\/em>
\nopenshift_metrics_cassandra_pvc_size: 40G<\/em>
\nopenshift_metrics_cassandra_request_cpu: 2000m<\/em>
\nopenshift_metrics_cassandra_request_memory: 2Gi<\/em>
\nopenshift_metrics_cassandra_storage_type: pv<\/em>
\nopenshift_metrics_cassandra_pvc_storage_class_name: ceph-storage
\nopenshift_metrics_cassanda_pvc_storage_class_name: ceph-storage<\/em>
\nopenshift_metrics_image_version: v3.9<\/em>
\nopenshift_metrics_install_metrics: True<\/em>
\nopenshift_metrics_duration: 14<\/em>
\nopenshift_metrics_hawkular_limits_cpu: 3000m<\/em>
\nopenshift_metrics_hawkular_limits_memory: 3Gi<\/em>
\nopenshift_metrics_hawkular_node_selector: {“region”:”infra”}<\/em>
\nopenshift_metrics_hawkular_requests_cpu: 2000m<\/em>
\nopenshift_metrics_hawkular_requests_memory: 2Gi<\/em>
\nopenshift_metrics_heapster_limits_cpu: 3000m<\/em>
\nopenshift_metrics_heapster_limits_memory: 3Gi<\/em>
\nopenshift_metrics_heapster_node_selector: {“region”:”infra”}<\/em>
\nopenshift_metrics_heapster_requests_cpu: 2000m<\/em>
\nopenshift_metrics_heapster_requests_memory: 2Gi<\/em><\/p><\/blockquote>\n

Note that we are defining both\u00a0openshift_metrics_cassandra_pvc_storage_class_name<\/em> and\u00a0openshit_metrics_cassanda_pvc_storage_class_name<\/em> due to a typo that was recently fixed<\/a>, yet not in OpenShift Origin last packages.<\/p>\n

Setting up those metrics may allow you to create Nagios commands based on querying for resources allocations and consumptions, using:<\/p>\n

$ oc adm top node –heapster-namespacce=openshift-infra –heapster-scheme=https node.example.com<\/em><\/p><\/blockquote>\n

 <\/p>\n

Another solution that integrates well with OpenShift is Prometheus, that could be deployed using the playbooks\/openshift-prometheus\/config.yml<\/em> playbook and those Ansible variables:<\/p>\n

\"Prometheus\"<\/a>

Prometheus showing OpenShift Pods CPU usages<\/p><\/div>\n

openshift_prometheus_alertbuffer_pvc_size: 20Gi<\/em>
\nopenshift_prometheus_alertbuffer_storage_class: ceph-storage<\/em>
\nopenshift_prometheus_alertbuffer_storage_type: pvc<\/em>
\nopenshift_prometheus_alertmanager_pvc_size: 20Gi<\/em>
\nopenshift_prometheus_alertmanager_storage_class: ceph-storage<\/em>
\nopenshift_prometheus_alertmanager_storage_type: pvc<\/em>
\nopenshift_prometheus_namespace: openshift-metrics<\/em>
\nopenshift_prometheus_node_selector: {“region”:”infra”}<\/em>
\nopenshift_prometheus_pvc_size: 20Gi<\/em>
\nopenshift_prometheus_state: present<\/em>
\nopenshift_prometheus_storage_class: ceph-storage<\/em>
\nopenshift_prometheus_storage_type: pvc<\/em><\/p><\/blockquote>\n

 <\/p>\n

We could also deploy some Grafana, that could include a pre-configured dashboard, rendering some Prometheus metrics – thanks to the playbooks\/openshift-grafana\/config.yml<\/em> playbook and the following Ansible variables:<\/p>\n

\"Grafana\"<\/a>

OpenShift Dashboard on Grafana<\/p><\/div>\n

openshift_grafana_datasource_name: prometheus<\/em>
\nopenshift_grafana_graph_granularity: 2m<\/em>
\nopenshift_grafana_namespace: openshift-grafana<\/em>
\nopenshift_grafana_node_exporter: True<\/em>
\nopenshift_grafana_node_selector: {“region”:”infra”}<\/em>
\nopenshift_grafana_prometheus_namespace: openshift-metrics<\/em>
\nopenshift_grafana_prometheus_serviceaccount: prometheus<\/em>
\nopenshift_grafana_storage_class: ceph-storage<\/em>
\nopenshift_grafana_storage_type: pvc<\/em>
\nopenshift_grafana_storage_volume_size: 15Gi<\/em><\/p><\/blockquote>\n

 <\/p>\n

And finally, we could also deploy logs centralization with the playbooks\/openshift-logging\/config.yml<\/em> playbook, setting the following:<\/p>\n

\"Kibana\"<\/a>

Kibana integration with EFK<\/p><\/div>\n

openshift_logging_install_logging: True
\nopenshift_logging_curator_default_days: ‘7’
\nopenshift_logging_curator_cpu_request: 100m
\nopenshift_logging_curator_memory_limit: 256Mi
\nopenshift_logging_curator_nodeselector: {“region”:”infra”}
\nopenshift_logging_elasticsearch_storage_type: pvc
\nopenshift_logging_es_cluster_size: ‘1’
\nopenshift_logging_es_cpu_request: ‘1’
\nopenshift_logging_es_memory_limit: 8Gi
\nopenshift_logging_es_pvc_storage_class_name: ceph-storage
\nopenshift_logging_es_pvc_dynamic: True
\nopenshift_logging_es_pvc_size: 25Gi
\nopenshift_logging_es_recover_after_time: 10m
\nopenshift_logging_es_nodeslector: {“region”:”infra”}
\nopenshift_logging_es_number_of_shards: ‘1’
\nopenshift_logging_es_number_of_replicas: ‘0’
\nopenshift_logging_fluentd_buffer_queue_limit: 1024
\nopenshift_logging_fluentd_buffer_size_limit: 1m
\nopenshift_logging_fluentd_cpu_request: 100m
\nopenshift_logging_fluentd_file_buffer_limit: 256Mi
\nopenshift_logging_fluentd_memory_limit: 512Mi
\nopenshift_logging_fluentd_nodeselector: {“region”:”infra”}
\nopenshift_logging_fluentd_replica_count: 2
\nopenshift_logging_kibana_cpu_request: 600m
\nopenshift_logging_kibana_hostname: kibana.router.intra.unetresgrossebite.com
\nopenshift_logging_kibana_memory_limit: 736Mi
\nopenshift_logging_kibana_proxy_cpu_request: 200m
\nopenshift_logging_kibana_proxy_memory_limit: 256Mi
\nopenshift_logging_kibana_replica_count: 2
\nopenshift_logging_kibana_nodeselector: {“region”:”infra”}<\/p><\/blockquote>\n

 <\/p>\n

Meanwhile we could note that cri-o is getting better support in the latter versions of OpenShift, among a never-ending list of ongoing works and upcoming features.<\/p>\n","protected":false},"excerpt":{"rendered":"

Today I am looking back on a few topics I had a hard time properly deploying using OpenShift 3.7 and missing proper dynamic provisioning despite a poorly-configured GlusterFS cluster. Since then, I deployed a 3 nodes Ceph cluster, using Sebastien Han’s ceph-ansible playbooks, allowing me to further experiment with persistent volumes. And OpenShift Origin 3.9 […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[8,7,13,6],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/764"}],"collection":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=764"}],"version-history":[{"count":8,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/764\/revisions"}],"predecessor-version":[{"id":778,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/764\/revisions\/778"}],"wp:attachment":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=764"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=764"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=764"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}