![\"After](\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2018\/01\/smallback-e1515265835587.jpg\")
<\/a>After replacing 4x MicroServer, by 3x SE318m1<\/p><\/div>\n
I’ve first tried setting my own, re-using my ProLian MicroServers. One of my master node was refusing to deploy, CPU usage averaging around 100%, systemctl consistently timing out while starting some process – that did start on my two other master nodes.
\nAfter trying to resize my KVMs in vain, I eventually went another way: shut down a stair of ProLian MicroServer, move them out of my rack and plug instead 3 servers I ordered a couple years ago, that never reached prod – due to doubts regarding overall power consumption, EDF being able to deliver enough Amperes, my switches not being able to provide with enough LACP channels, my not having enough SSDs or quad-port Ethernet cards in stock to fill these servers,\u00a0 …<\/p>\n
I eventually compromised, and harvested any 500G SSDs disks available out of my Ceph cluster, mounting one per 1U server.<\/p>\n
Final setup involves the following physical servers:<\/p>\n
- \n
- a custom tower (core i5, 32G DDR, 128G SSD disk)<\/li>\n
- \n3x HP SE316M1 (2xE5520, 24G DDR) – 500G SSD<\/div>\n<\/li>\n
- \n
2x HP SE1102 (2xE5420 12G DDR) – 500G SSD<\/div>\n<\/li>\n- \n
3x ProLian MicroServer G5 (Turion, 4-8G DDR) – 64G SSD + 3×3-4T HDD<\/div>\n<\/li>\n<\/ul>\nAnd on top of these, a set of KVM instances, including:<\/p>\n
- \n
- 3 master nodes (2 CPU, 8G RAM)<\/li>\n
- 3 infra nodes (2 CPU, 6G RAM)<\/li>\n
- 3 compute nodes (4 CPU, 10G RAM @SE316M1)<\/li>\n
- 3 storage nodes (1 CPU, 3G RAM @MicroServer)<\/li>\n<\/ul>\n
Everything running on CentOS7. Except for some Ansible DomU I would use deploying OpenShift, running Debian Stretch.<\/p>\n
<\/p>\n
OpenShift can be deployed using Ansible<\/a>. And as I’ve been writing my own roles for the past couple years, I can testify these ones are amazing.<\/p>\n
![\"GlusterFS](\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2018\/01\/oshift-glusterfs2-300x192.png\")
<\/a>GlusterFS @OpenShift<\/p><\/div>\n
First ansible run would be done setting the following variables, bootstrapping service on top of my existing domain name, and LDAP server.<\/p>\n
ansible_ssh_user: root
\nopenshift_deployment_type: origin
\nopenshift_disable_check: disk_availability,docker_storage,memory_availability
\nopenshift_master_cluster_method: native
\nopenshift_master_cluster_hostname: openshift.intra.unetresgrossebite.com
\nopenshift_master_cluster_public_hostname: openshift.intra.unetresgrossebite.com
\nopenshift_master_default_subdomain: router.intra.unetresgrossebite.com
\nopenshift.common.dns_domain: openshift.intra.unetresgrossebite.com
\nopenshift_clock_enabled: True
\nopenshift_node_kubelet_args: {‘pods-per-core’: [’10’], ‘max-pods’: [‘250’], ‘image-gc-high-threshold’: [’90’], ‘image-gc-low-threshold’: [’80’]}
\nopenshift_master_identity_providers:
\n– name: UneTresGrosseBite
\n challenge: ‘true’
\n login: ‘true’
\n kind: LDAPPasswordIdentityProvider
\n attributes:
\n id: [‘dn’]
\n email: [‘mail’]
\n name: [‘sn’]
\n preferredUsername: [‘uid’]
\n bindDN: cn=openshift,ou=services,dc=unetresgrossebite,dc=com
\n bindPassword: secret
\n ca: ldap-chain.crt
\n insecure: ‘false’
\n url: ‘ldaps:\/\/netserv.vms.intra.unetresgrossebite.com\/ou=users,dc=unetresgrossebite,dc=com?uid?sub?(&(objectClass=inetOrgPerson)(!(pwdAccountLockedTime=*)))’
\nopenshift_master_ldap_ca_file: \/root\/ldap-chain.crt<\/p><\/blockquote>\nSetting up glusterfs, note you may have difficulties setting gluster block devices as group vars, and could find a solution sticking to defining these directly into your inventory file:<\/p>\n
[glusterfs]
\ngluster1.friends.intra.unetresgrossebite.com glusterfs_ip=10.42.253.100 glusterfs_devices='[ “\/dev\/vdb”, “\/dev\/vdc”, “\/dev\/vdd” ]’
\ngluster2.friends.intra.unetresgrossebite.com glusterfs_ip=10.42.253.101 glusterfs_devices='[ “\/dev\/vdb”, “\/dev\/vdc”, “\/dev\/vdd” ]’
\ngluster3.friends.intra.unetresgrossebite.com glusterfs_ip=10.42.253.102 glusterfs_devices='[ “\/dev\/vdb”, “\/dev\/vdc”, “\/dev\/vdd” ]’<\/p><\/blockquote>\nApply the main playbook with:<\/p>\n
ansible-playbook playbooks\/byo\/config.yml -i .\/hosts<\/p><\/blockquote>\n
Have a break: with 4 CPUs & 8G RAM on my ansible host, applying a single variable change (pretty much everything was installed beforehand), I would still need over an hour and a half applying the full playbook:\u00a0whenever possible, stick to whatever service-specific playbook you may find, …<\/p>\n
![\"Jenkins](\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2018\/01\/oshift-jenkins-252x300.png\")
<\/a>Jenkins @OpenShift<\/p><\/div>\n
As a sidenote, be careful to properly set your domain name before deploying glusterfs. So far, while I was able to update my domain name almost everywhere running Ansible playbooks back, GlusterFS’s hekiti route was the first I noticed not being renamed.
\nShould you fuck up your setup, you can use oc project glusterfs<\/em> then oc get pods<\/em> to locate your running containers, use oc rsh <container><\/em> then rm -fr \/var\/lib\/hekiti<\/em> to purge stuff that may prevent further deployments, …
\nThen\u00a0oc delete project glusterfs<\/em>, to purge almost everything else.
\nYou may also use running\u00a0docker images | grep gluster<\/em> and docker rmi <images><\/em>, … As well as making sure to wipe the first sectors of your gluster disks (for d in b c d; do dd if=\/dev\/zero of=\/dev\/vd$d bs=1M count=8; done<\/em>). You may need to reboot your hosts (if a wipefs -a \/dev\/drive<\/em> returns with an error). Finally, re-deploy a new GlusterFS cluster from scratch using Ansible.<\/p>\n<\/p>\n
Once done with the main playbook, you should be able to log into your OpenShift dashboard. Test it by deploying Jenkins.<\/p>\n
![\"hawkular](\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2018\/01\/openshift-hawkular-300x90.png\")
<\/a>Hawkular integration @OpenShift<\/p><\/div>\n
<\/p>\n
<\/p>\n
You could (should) also look into deploying OpenShift cluster metrics collection, based on Hawkular & Heapster.
\nSticking with volatile storage, you would need adding the following variable to all your hosts:<\/p>\n<\/p>\n
openshift_metrics_install_metrics: True<\/p><\/blockquote>\n
Note to deploy these roles, you would have to install on your Ansible host (manually!) python-passlib<\/em>, apache2-utils<\/em> and\u00a0openjdk-8-jdk-headless<\/em> (assuming Debian\/Ubuntu). You may then deploy metrics using the\u00a0playbooks\/byo\/openshift-cluster\/openshift-metrics.yml<\/em> playbook.<\/p>\n
Hawkular integration would allow you to track resources usage directly from OpenShift dashboard.<\/p>\n
- \n