Last week, one of my five Ceph hosts was unreachable.
\nInvestigating, I noticed the OSD daemons were still running. Only daemons using the root file system, where either crashed (the local ceph MON daemon) or unable to process requests (SSH daemon was still answering, cleanly closing\u00a0the connection).<\/p>\n
After rebooting the system and looking at logs, I could see a lot of I\/O errors. I left the console logged in to root, waiting for the next occurrence.
\nHaving no spare 60GB SSD, I ordered one.<\/p>\n
Two days later, the same problem occurred. From the console, I was unable to run anything (mostly segfaults and ENOENT).
\nAgain, I was able to reboot. This time, I dropped a couple LVMs, unmounted the swap partition, and resized my VG to make sure I had a fair amount of unallocated space on my faulty disk.<\/p>\n
The problem persisted, while average uptime was significantly getting lower.
\nI progressively disabled local OSSEC daemon, puppet, a few crontabs, collectd, munin, … only keeping ceph, nagios and ssh running. The problem kept happening, every 12 to 48 hours.<\/p>\n
This morning, the server wasn’t even able to\u00a0boot.
\nChecking the BIOS, my root SSD wasn’t detected.
\nAttaching it to some USB dock, I had to wait a couples minutes before the disk was actually detected by my laptop\u00a0(Ubuntu 14.04.02), and my desktop (Debian 7.8).
\nI caught a break when receiving my new disk at\u00a011 AM.
\nRunning dd from the faulty disk to the new one took around 50 minutes (20MB\/s, I can’t believe it!).
\nSyncing (1x512G SSD, 2*4T & 1*3T HDD) after 8\u00a0hours of downtime, took around\u00a0half an hour. Knowing I run a fairly busy mail server, some\u00a0nntp\u00a0index, …), this is a new tangible improvement brought by Hammer, over Firefly.<\/p>\n
I’m now preparing to send\u00a0the faulty disk to my re-seller, for replacement. At least, I would have one handy, for the next failure.<\/p>\n
Morality: cheap is unreasonable. Better be lucky.<\/p>\n","protected":false},"excerpt":{"rendered":"
Last week, one of my five Ceph hosts was unreachable. Investigating, I noticed the OSD daemons were still running. Only daemons using the root file system, where either crashed (the local ceph MON daemon) or unable to process requests (SSH daemon was still answering, cleanly closing\u00a0the connection). After rebooting the system and looking at logs, […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[8,5,2],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/128"}],"collection":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=128"}],"version-history":[{"count":3,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/128\/revisions"}],"predecessor-version":[{"id":131,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=\/wp\/v2\/posts\/128\/revisions\/131"}],"wp:attachment":[{"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=128"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=128"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.unetresgrossebite.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=128"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}