Munin for Hadoop

Configuration of Munin on Hadoop cluster:

For each of the processes to monitor, you should add: (you should find them in hadoop-env.sh)

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2101 -Dcom.sun.management.jmxremote.ssl=false -  Dcom.sun.management.jmxremote.authenticate=false -Dja
va.rmi.server.hostname=10.239.24.12 $HADOOP_NAMENODE_OPTS"

export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2102 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=f
alse -Djava.rmi.server.hostname=10.239.24.12 $HADOOP_SECONDARYNAMENODE_OPTS"

export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2103 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dja
va.rmi.server.hostname=10.239.24.13 $HADOOP_DATANODE_OPTS"

export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2104 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dja
va.rmi.server.hostname=10.239.24.13 $HADOOP_BALANCER_OPTS"

export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2105 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -D
java.rmi.server.hostname=10.239.24.13 $HADOOP_JOBTRACKER_OPTS"

export HADOOP_TASKTRACKER_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2106 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -
Djava.rmi.server.hostname=10.239.24.13 $HADOOP_TASKTRACKER_OPTS"

install munin on master with: (or equivalent, depends of the Linux distribution)

sudo yum install munin 

install munin-node on each slave with:

sudo yum install munin-node 

Add jmx_ and jmxquery.jar in

/usr/share/munin/plugins

Create symlinks in /etc/munin/plugins/

jmx_hadoop -> /usr/share/munin/plugins/jmx_

Everything after jmx_ specifies the name of a config file which defines what you want to graph.

This plugin expects those configs to be in /etc/munin/jmx-conf.d. In addition you have to configure the plugin so it knows where to find jmx. In this case:

/etc/munin/jmx-conf.d contains: 
hadoop.conf

/etc/munin/plugin-conf.d/ contains
hadoop 

which looks like this:

\[jmx_hadoop*\]
env.category jvm
env.username monitor
env.password monitor
env.JRE_HOME /usr/lib/jvm/java-6-sun/jre
env.jmxurl service:jmx:rmi:///jndi/rmi://localhost:2103/jmxrm

Useful commands:

/etc/init.d/munin-node restart 
chkconfig munin-node on
/usr/bin/munin-cron 

Remove windows characters to file

sed -i 's/\r//' /usr/share/munin/plugins/jmx_

Check logs:

cd /var/log/munin/ 
cd /var/www/html/munin/ 
check munin-{var/log/update}.loge}.log

Thanks a lot to Stuart Teasdale for his precious help on installation and configuration!

You will find a link to jmx_ [1] And to jmxquery.jar [2]

Keren Ouaknine