{"id":205,"date":"2015-08-24T16:55:58","date_gmt":"2015-08-24T14:55:58","guid":{"rendered":"https:\/\/blog.unetresgrossebite.com\/?p=205"},"modified":"2015-08-30T15:31:49","modified_gmt":"2015-08-30T13:31:49","slug":"scaling-out-with-ceph","status":"publish","type":"post","link":"https:\/\/blog.unetresgrossebite.com\/?p=205","title":{"rendered":"Scaling out with Ceph"},"content":{"rendered":"

A few months ago, I installed a Ceph cluster hosting disk images, for my OpenNebula cloud.
\nThis cluster is based on 5 ProLian N54L, each with a 60G SSD\u00a0for the main filesystems, some with 1 512G SSD OSD, all with 3 disk drives from 1 to 4T. SSD are grouped in a pool, HDD in an other.<\/p>\n

\"OpenNebula<\/a>

OpenNebula Datastores View, having 5 Ceph OSD hosts<\/p><\/div>\n

Now that most my services are in this cluster, I’m left with very few free space.
\nThe good news is there is no significant impact on performances, as I was experiencing with ZFS.
\nThe bad news, is that I urgently need to add some storage space.<\/p>\n

Last Sunday, I ordered my sixth\u00a0N54L on eBay (from my “official” refurbish-er, BargainHardware<\/a>) and a few disks.
\nAfter receiving everything, I installed the latest Ubuntu LTS (Trusty) from my PXE, installed puppet, prepared everything, … In about an hour, I was ready to add my disks.<\/p>\n

I use\u00a0a custom crush map, and the\u00a0osd “crush update on start<\/em>” set to false, in my ceph.conf.
\nThis was the first time I tested this, and I was pleased to see I can run ceph-deploy<\/em> to prepare my OSD, without automatically adding it to the default CRUSH root – especially having two pools.
\nFrom my ceph-deploy<\/em> host (some Xen PV\u00a0I use hosting ceph-dash,\u00a0munin and nagios probes related to ceph, but with no OSD nor MON actually running), I ran the following:<\/p>\n

# ceph-deploy install erebe
\n# ceph-deploy disk list erebe
\n# ceph-deploy disk zap erebe:sda
\n# ceph-deploy disk zap erebe:sdb
\n# ceph-deploy disk zap erebe:sdc
\n# ceph-deploy disk zap erebe:sdd
\n# ceph-deploy osd prepare erebe:sda
\n# ceph-deploy osd prepare erebe:sda
\n# ceph-deploy osd prepare erebe:sdb
\n# ceph-deploy osd prepare erebe:sdc
\n# ceph-deploy osd prepare erebe:sdd<\/p><\/blockquote>\n

At that point, the 4 new OSD were up and running according to ceph status, though no data was assigned to them.
\nNext step was to update my crushmap, including these new OSDs in the proper root.<\/p>\n

# ceph osd getcrushmap -o compiled_crush
\n# crushtool -d compiled_crush -o plain_crush
\n# vi plain_crush
\n# crushtool -c plain_crush -o new-crush
\n# ceph osd setcrushmap -i new-crush<\/p><\/blockquote>\n

For the record, the content of my current crush map is the following:
\n# begin crush map
\ntunable choose_local_tries 0
\ntunable choose_local_fallback_tries 0
\ntunable choose_total_tries 50
\ntunable chooseleaf_descend_once 1
\ntunable straw_calc_version 1
\n# devices
\ndevice 0 osd.0
\ndevice 1 osd.1
\ndevice 2 osd.2
\ndevice 3 osd.3
\ndevice 4 osd.4
\ndevice 5 osd.5
\ndevice 6 osd.6
\ndevice 7 osd.7
\ndevice 8 osd.8
\ndevice 9 osd.9
\ndevice 10 osd.10
\ndevice 11 osd.11
\ndevice 12 osd.12
\ndevice 13 osd.13
\ndevice 14 osd.14
\ndevice 15 osd.15
\ndevice 16 osd.16
\ndevice 17 osd.17
\ndevice 18 osd.18
\ndevice 19 osd.19
\ndevice 20 osd.20
\ndevice 21 osd.21
\n# types
\ntype 0 osd
\ntype 1 host
\ntype 2 chassis
\ntype 3 rack
\ntype 4 row
\ntype 5 pdu
\ntype 6 pod
\ntype 7 room
\ntype 8 datacenter
\ntype 9 region
\ntype 10 root
\n# buckets
\nhost nyx-hdd {
\nid -2 # do not change unnecessarily
\n# weight 10.890
\nalg straw
\nhash 0 # rjenkins1
\nitem osd.1 weight 3.630
\nitem osd.2 weight 3.630
\nitem osd.3 weight 3.630
\n}
\nhost eos-hdd {
\nid -3 # do not change unnecessarily
\n# weight 11.430
\nalg straw
\nhash 0 # rjenkins1
\nitem osd.5 weight 3.630
\nitem osd.6 weight 3.900
\nitem osd.7 weight 3.900
\n}
\nhost hemara-hdd {
\nid -4 # do not change unnecessarily
\n# weight 9.980
\nalg straw
\nhash 0 # rjenkins1
\nitem osd.9 weight 3.630
\nitem osd.10 weight 3.630
\nitem osd.11 weight 2.720
\n}
\nhost selene-hdd {
\nid -5 # do not change unnecessarily
\n# weight 5.430
\nalg straw
\nhash 0 # rjenkins1
\nitem osd.12 weight 1.810
\nitem osd.13 weight 2.720
\nitem osd.14 weight 0.900
\n}
\nhost helios-hdd {
\nid -6 # do not change unnecessarily
\n# weight 3.050
\nalg straw
\nhash 0 # rjenkins1
\nitem osd.15 weight 1.600
\nitem osd.16 weight 0.700
\nitem osd.17 weight 0.750
\n}
\nhost erebe-hdd {
\nid -7 # do not change unnecessarily
\n# weight 7.250
\nalg straw
\nhash 0 # rjenkins1
\nitem osd.19 weight 2.720
\nitem osd.20 weight 1.810
\nitem osd.21 weight 2.720
\n}
\nroot hdd {
\nid -1 # do not change unnecessarily
\n# weight 40.780
\nalg straw
\nhash 0 # rjenkins1
\nitem nyx-hdd weight 10.890
\nitem eos-hdd weight 11.430
\nitem hemara-hdd weight 9.980
\nitem selene-hdd weight 5.430
\nitem helios-hdd weight 3.050
\nitem erebe-hdd weight 7.250
\n}
\nhost nyx-ssd {
\nid -42 # do not change unnecessarily
\n# weight 0.460
\nalg straw
\nhash 0 # rjenkins1
\nitem osd.0 weight 0.460
\n}
\nhost eos-ssd {
\nid -43 # do not change unnecessarily
\n# weight 0.460
\nalg straw
\nhash 0 # rjenkins1
\nitem osd.4 weight 0.460
\n}
\nhost hemara-ssd {
\nid -44 # do not change unnecessarily
\n# weight 0.450
\nalg straw
\nhash 0 # rjenkins1
\nitem osd.8 weight 0.450
\n}
\nhost erebe-ssd {
\nid -45 # do not change unnecessarily
\n# weight 0.450
\nalg straw
\nhash 0 # rjenkins1
\nitem osd.18 weight 0.450
\n}
\nroot ssd {
\nid -41 # do not change unnecessarily
\n# weight 3.000
\nalg straw
\nhash 0 # rjenkins1
\nitem nyx-ssd weight 1.000
\nitem eos-ssd weight 1.000
\nitem hemara-ssd weight 1.000
\nitem erebe-ssd weight 1.000
\n}
\n# rules
\nrule hdd {
\nruleset 0
\ntype replicated
\nmin_size 1
\nmax_size 10
\nstep take hdd
\nstep chooseleaf firstn 0 type host
\nstep emit
\n}
\nrule ssd {
\nruleset 1
\ntype replicated
\nmin_size 1
\nmax_size 10
\nstep take ssd
\nstep chooseleaf firstn 0 type host
\nstep emit
\n}
\n# end crush map<\/code><\/p>\n

Applying the new crush map, a 20 hours process started, moving placement groups.<\/p>\n

\"OpenNebula<\/a>

OpenNebula Datastores View, having 6 Ceph OSD hosts<\/p><\/div>\n

# ceph-diskspace
\n\/dev\/sdc1 3.7T 2.0T 1.7T 55% \/var\/lib\/ceph\/osd\/ceph-6
\n\/dev\/sda1 472G 330G 143G 70% \/var\/lib\/ceph\/osd\/ceph-4
\n\/dev\/sdb1 3.7T 2.4T 1.4T 64% \/var\/lib\/ceph\/osd\/ceph-5
\n\/dev\/sdd1 3.7T 2.4T 1.4T 65% \/var\/lib\/ceph\/osd\/ceph-7
\n\/dev\/sda1 442G 329G 114G 75% \/var\/lib\/ceph\/osd\/ceph-18
\n\/dev\/sdb1 2.8T 2.1T 668G 77% \/var\/lib\/ceph\/osd\/ceph-19
\n\/dev\/sdc1 1.9T 1.3T 593G 69% \/var\/lib\/ceph\/osd\/ceph-20
\n\/dev\/sdd1 2.8T 2.0T 808G 72% \/var\/lib\/ceph\/osd\/ceph-21
\n\/dev\/sdc1 927G 562G 365G 61% \/var\/lib\/ceph\/osd\/ceph-17
\n\/dev\/sdb1 927G 564G 363G 61% \/var\/lib\/ceph\/osd\/ceph-16
\n\/dev\/sda1 1.9T 1.2T 630G 67% \/var\/lib\/ceph\/osd\/ceph-15
\n\/dev\/sdb1 3.7T 2.8T 935G 75% \/var\/lib\/ceph\/osd\/ceph-9
\n\/dev\/sdd1 2.8T 1.4T 1.4T 50% \/var\/lib\/ceph\/osd\/ceph-11
\n\/dev\/sda1 461G 274G 187G 60% \/var\/lib\/ceph\/osd\/ceph-8
\n\/dev\/sdc1 3.7T 2.2T 1.5T 60% \/var\/lib\/ceph\/osd\/ceph-10
\n\/dev\/sdc1 3.7T 1.9T 1.8T 52% \/var\/lib\/ceph\/osd\/ceph-1
\n\/dev\/sde1 3.7T 2.0T 1.7T 54% \/var\/lib\/ceph\/osd\/ceph-3
\n\/dev\/sdd1 3.7T 2.3T 1.5T 62% \/var\/lib\/ceph\/osd\/ceph-2
\n\/dev\/sdb1 472G 308G 165G 66% \/var\/lib\/ceph\/osd\/ceph-0
\n\/dev\/sdb1 1.9T 1.2T 673G 64% \/var\/lib\/ceph\/osd\/ceph-12
\n\/dev\/sdd1 927G 580G 347G 63% \/var\/lib\/ceph\/osd\/ceph-14
\n\/dev\/sdc1 2.8T 2.0T 813G 71% \/var\/lib\/ceph\/osd\/ceph-13<\/p><\/blockquote>\n

I’m really satisfied by the way ceph is constantly improving their product.
\nHaving discussed with several interviewers in the last few weeks, I’m still having to explain why ceph rbd is not to be confused with cephfs, and if the latter may not be production ready, rados storage is just the thing you could be looking for distributing your storage.<\/p>\n","protected":false},"excerpt":{"rendered":"

A few months ago, I installed a Ceph cluster hosting disk images, for my OpenNebula cloud. This cluster is based on 5 ProLian N54L, each with a 60G SSD\u00a0for the main filesystems, some with 1 512G SSD OSD, all with 3 disk drives from 1 to 4T. SSD are grouped in a pool, HDD in […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[5,2],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/205"}],"collection":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=205"}],"version-history":[{"count":11,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/205\/revisions"}],"predecessor-version":[{"id":221,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/205\/revisions\/221"}],"wp:attachment":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=205"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=205"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=205"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}