{"id":590,"date":"2017-07-21T18:51:38","date_gmt":"2017-07-21T16:51:38","guid":{"rendered":"https:\/\/blog.unetresgrossebite.com\/?p=590"},"modified":"2017-12-21T18:39:40","modified_gmt":"2017-12-21T17:39:40","slug":"ceph-luminous","status":"publish","type":"post","link":"https:\/\/blog.unetresgrossebite.com\/?p=590","title":{"rendered":"Ceph Luminous &#8211; 12"},"content":{"rendered":"<p>In the last few days, Ceph published Luminous 12.1.1 packages to their repositories, release candidate of their future LTS. Having had bad experiences with their previous RC, I gave it a fresh look, dropping ceph-deploy and writing my own Ansible roles instead.<\/p>\n<div id=\"attachment_594\" style=\"width: 310px\" class=\"wp-caption alignright\"><a href=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/luminoushealth.png\"><img aria-describedby=\"caption-attachment-594\" decoding=\"async\" loading=\"lazy\" class=\"size-medium wp-image-594\" src=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/luminoushealth-300x134.png\" alt=\"typo listing RGW process\" width=\"300\" height=\"134\" srcset=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/luminoushealth-300x134.png 300w, https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/luminoushealth.png 452w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-594\" class=\"wp-caption-text\">Small typo displaying Ceph Health, if you can notice it<\/p><\/div>\n<p>Noticeable changes since Luminous include CephFS being -allegedly- stable. I didn&#8217;t test it myself yet, although I&#8217;ve been hearing about that feature being unstable since my first days testing Ceph, years ago.<\/p>\n<div id=\"attachment_602\" style=\"width: 165px\" class=\"wp-caption alignleft\"><a href=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/radosgw-multisite-status.png\"><img aria-describedby=\"caption-attachment-602\" decoding=\"async\" loading=\"lazy\" class=\"size-medium wp-image-602\" src=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/radosgw-multisite-status-155x300.png\" alt=\"RadosGateway multisite sync status\" width=\"155\" height=\"300\" srcset=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/radosgw-multisite-status-155x300.png 155w, https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/radosgw-multisite-status.png 305w\" sizes=\"(max-width: 155px) 100vw, 155px\" \/><\/a><p id=\"caption-attachment-602\" class=\"wp-caption-text\">RadosGateway multisite sync status<\/p><\/div>\n<p>Another considerable change that showed up and is now considered stable, is a replacement implementation of Ceph FileStore (relying on ext4, xfs or btrfs partitions), called BlueStore. The main change being that Ceph Object Storage processes would no longer mount a large filesystem storing their data. Beware that recovery scripts reconstructing block devices scanning for PG content in OSD filesystems would no longer work &#8211; it is yet unclear how a disaster recovery would work, recovering data from an offline cluster. No surprises otherwise, so far so good.<\/p>\n<p>Also advertised: the RBD-mirror daemon (introduced lately) is now considered stable running in HA. From what I&#8217;ve seen, it is yet unclear how to configure HA starting several mirrors in a single cluster &#8211; I very much doubt this would work out of the box. We&#8217;ll probably have to wait a little longer, for Ceph documentation to reflect the last changes introduced on that matter.<\/p>\n<p>As I had 2 Ceph clusters, I could confirm RadosGW Multisite configuration works perfectly. Now that&#8217;s not a new feature, still it&#8217;s the first time I actually set this up. Buckets are eventually replicated to remote cluster. <a href=\"http:\/\/docs.ceph.com\/docs\/master\/radosgw\/multisite\" target=\"_blank\" rel=\"noopener\">Documentation is way more exhaustive<\/a>, works as advertised: I&#8217;ll stick to this, until we learn more about RBD mirroring.<\/p>\n<div id=\"attachment_599\" style=\"width: 310px\" class=\"wp-caption alignright\"><a href=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/restapi.png\"><img aria-describedby=\"caption-attachment-599\" decoding=\"async\" loading=\"lazy\" class=\"size-medium wp-image-599\" src=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/restapi-300x130.png\" alt=\"Querying for the MON commands\" width=\"300\" height=\"130\" srcset=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/restapi-300x130.png 300w, https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/restapi-768x334.png 768w, https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/restapi-1024x445.png 1024w, https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/restapi.png 1029w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-599\" class=\"wp-caption-text\">Ceph RestAPI Gateway<\/p><\/div>\n<p>Freshly introduced: the Ceph RestAPI Gateway. Again, we&#8217;re missing some docs yet. On paper, this service should allow you to query your cluster as you would have with Ceph CLI tools, via a Restful API. Having set one up, it isn&#8217;t much complicated &#8211; I would recommend not to use their built-in webserver, and instead use nginx and uwsgi. The basics on that matter could be found <a href=\"https:\/\/gist.github.com\/jharley\/e23034135966bb7b438e\">on GitHub<\/a>.<\/p>\n<div id=\"attachment_605\" style=\"width: 310px\" class=\"wp-caption alignleft\"><a href=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-pg-cleanup.png\"><img aria-describedby=\"caption-attachment-605\" decoding=\"async\" loading=\"lazy\" class=\"size-medium wp-image-605\" src=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-pg-cleanup-300x104.png\" alt=\"Ceph health turns to warning, watch for un-scrubbed PGs\" width=\"300\" height=\"104\" srcset=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-pg-cleanup-300x104.png 300w, https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-pg-cleanup.png 447w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-605\" class=\"wp-caption-text\">Ceph health turns to warning, watch for un-scrubbed PGs<\/p><\/div>\n<p>Even though Ceph Luminous shouldn&#8217;t reach LTS before their 12.2.0 release, as of today, I can confirm Debian Stretch packages are working relatively well on a 3-MON 3-MGR 3-OSD 2-RGW with some haproxy balancer setup, serving with s3-like buckets. Although note there is some weirdness regarding PG scrubbing, you may need to add a cron job &#8230; And if you consider running Ceph on commodity hardware, consider that their last releases <a href=\"http:\/\/tracker.ceph.com\/issues\/20529\" target=\"_blank\" rel=\"noopener\">may be broken<\/a>.<\/p>\n<div id=\"attachment_707\" style=\"width: 310px\" class=\"wp-caption alignright\"><a href=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-dashboard.png\"><img aria-describedby=\"caption-attachment-707\" decoding=\"async\" loading=\"lazy\" class=\"size-medium wp-image-707\" src=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-dashboard-300x149.png\" alt=\"ceph dashboard\" width=\"300\" height=\"149\" srcset=\"https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-dashboard-300x149.png 300w, https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-dashboard-768x381.png 768w, https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-dashboard-1024x508.png 1024w, https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-dashboard-720x360.png 720w, https:\/\/blog.unetresgrossebite.com\/wp-content\/uploads\/2017\/07\/ceph-dashboard.png 1366w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-707\" class=\"wp-caption-text\">Ceph Dashboard<\/p><\/div>\n<p>edit: LTS released as of late August: SSE4.2 support still mandatory deploying your Luminous cluster, although <a href=\"https:\/\/github.com\/ceph\/ceph\/pull\/17388\/files\" target=\"_blank\" rel=\"noopener\">a fix recently reached their master branch<\/a>, &#8230;.<\/p>\n<p>As of late September, Ceph 12.2.1 release can actually be installed on older, commodity hardware.<br \/>\nMeanwhile, a few screenshots of <a href=\"http:\/\/ceph.com\/community\/new-luminous-dashboard\/\" target=\"_blank\" rel=\"noopener\">Ceph Dashboard were posted to ceph website<\/a>, advertising on that new feature.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In the last few days, Ceph published Luminous 12.1.1 packages to their repositories, release candidate of their future LTS. Having had bad experiences with their previous RC, I gave it a fresh look, dropping ceph-deploy and writing my own Ansible roles instead. Noticeable changes since Luminous include CephFS being -allegedly- stable. I didn&#8217;t test it [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[5,10,2],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/590"}],"collection":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=590"}],"version-history":[{"count":20,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/590\/revisions"}],"predecessor-version":[{"id":708,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/590\/revisions\/708"}],"wp:attachment":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=590"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=590"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=590"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}