Munin for Hadoop
Configuration of Munin on Hadoop cluster:
For each of the processes to monitor, you should add: (you should find them in hadoop-env.sh)
# Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2101 -Dcom.sun.management.jmxremote.ssl=false - Dcom.sun.management.jmxremote.authenticate=false -Dja va.rmi.server.hostname=10.239.24.12 $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2102 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=f alse -Djava.rmi.server.hostname=10.239.24.12 $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2103 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dja va.rmi.server.hostname=10.239.24.13 $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2104 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dja va.rmi.server.hostname=10.239.24.13 $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2105 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -D java.rmi.server.hostname=10.239.24.13 $HADOOP_JOBTRACKER_OPTS"
export HADOOP_TASKTRACKER_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2106 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false - Djava.rmi.server.hostname=10.239.24.13 $HADOOP_TASKTRACKER_OPTS"
install munin on master with:
(or equivalent, depends of the Linux distribution)
sudo yum install munin
install munin-node on each slave with:
sudo yum install munin-node
Add jmx_ and jmxquery.jar in
/usr/share/munin/plugins
Create symlinks in /etc/munin/plugins/
jmx_hadoop -> /usr/share/munin/plugins/jmx_
Everything after jmx_ specifies the name of a config file which defines what you want to graph.
This plugin expects those configs to be in /etc/munin/jmx-conf.d. In addition you have to configure the plugin so it knows where to find jmx. In this case:
/etc/munin/jmx-conf.d contains: hadoop.conf
/etc/munin/plugin-conf.d/ contains hadoop
which looks like this:
\[jmx_hadoop*\] env.category jvm env.username monitor env.password monitor env.JRE_HOME /usr/lib/jvm/java-6-sun/jre env.jmxurl service:jmx:rmi:///jndi/rmi://localhost:2103/jmxrm
Useful commands:
/etc/init.d/munin-node restart chkconfig munin-node on /usr/bin/munin-cron
Remove windows characters to file
sed -i 's/\r//' /usr/share/munin/plugins/jmx_
Check logs:
cd /var/log/munin/
cd /var/www/html/munin/
check munin-{var/log/update}.loge}.log
Thanks a lot to Stuart Teasdale for his precious help on installation and configuration!
You will find a link to jmx_ here And to jmxquery.jar here
Keren Ouaknine
