Munin is a popular supervision solution mostly based on perl.<\/p>\n
As of most supervision and monitoring solution, munin relies on an agent you would install on hosts to track metrics from, and a server gathering metrics and serving them via CGI, generating graphics from collected data.
\nAs of most supervision and monitoring solutions, munin uses loadable probes (plugins) to extract metrics.<\/p>\n
The server side consists of a crontab. Iterating on all your hosts, it would list and execute all available probes, store the results into RRD.
\nOver thousand hosts, you may need to export munin working root to some tmpfs storage, adding a crontab to regularly rsync the content from this tmpfs to a persistent directory. In any case, hosting your filesystem on some SSD is relevant.<\/p>\n
The main advantage of munin over its alternatives, I think, is its simplicity.
\nMain concurrent here being Collectd, which default web front end is particularly ugly and unpractical. And alternate front ends such as SickMuse are nice-looking, but still lack from simplicity, while some require excessively redundant configuration before being able to present something comparable to munin, in terms of relevance and efficiency.<\/p>\n
It is also very easy, to implement your own probes. From scripts to binaries, with a simple way to set contextual variables – evaluated when the plugin name matches the context expression – and requiring a single declaration on the server to collect all metrics served by a node: I don’t know of any supervision solution easier to deploy. And to incorporate to your orchestration infrastructure.<\/p>\n
Finally, debugging probes is conceptually easy: all you need is telnet.<\/p>\n
muninone:~# telnet 10.42.242.98 4949
\nTrying 10.42.242.98...
\nConnected to 10.42.242.98.
\nEscape character is '^]'.
\n# munin node at cronos.adm.intra.unetresgrossebite.com
\nlist
\ncpu df load memory ntp_kernel_err ntp_kernel_pll_freq ntp_kernel_pll_off open_files processes users vmstat
\nfetch cpu
\nuser.value 3337308
\nnice.value 39168
\nsystem.value 13389003
\ninterrupt.value 6679408
\nidle.value 701005749
\n.
\nquit
\nConnection closed by foreign host.
\nmuninone:~#<\/code><\/p>\n