Munin for Hadoop

From Lawa
Jump to: navigation, search

Configuration of Munin on Hadoop cluster:

For each of the processes to monitor, you should add: (you should find them in hadoop-env.sh) 

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2101 -Dcom.sun.management.jmxremote.ssl=false -  Dcom.sun.management.jmxremote.authenticate=false -Dja
va.rmi.server.hostname=10.239.24.12 $HADOOP_NAMENODE_OPTS"

export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2102 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=f
alse -Djava.rmi.server.hostname=10.239.24.12 $HADOOP_SECONDARYNAMENODE_OPTS"

export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2103 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dja
va.rmi.server.hostname=10.239.24.13 $HADOOP_DATANODE_OPTS"

export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2104 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dja
va.rmi.server.hostname=10.239.24.13 $HADOOP_BALANCER_OPTS"

export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2105 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -D
java.rmi.server.hostname=10.239.24.13 $HADOOP_JOBTRACKER_OPTS"

export HADOOP_TASKTRACKER_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2106 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -
Djava.rmi.server.hostname=10.239.24.13 $HADOOP_TASKTRACKER_OPTS"


install munin on master with: (or equivalent, depends of the Linux distribution) 

sudo yum install munin

install munin-node on each slave with: 

sudo yum install munin-node

Add jmx_ and jmxquery.jar in 

/usr/share/munin/plugins

Create symlinks in /etc/munin/plugins/ 

jmx_hadoop -> /usr/share/munin/plugins/jmx_

Everything after jmx_ specifies the name of a config file which defines what you want to graph.

This plugin expects those configs to be in /etc/munin/jmx-conf.d. In addition you have to configure the plugin so it knows where to find jmx. In this case: 

/etc/munin/jmx-conf.d contains: 
hadoop.conf

/etc/munin/plugin-conf.d/ contains
hadoop

which looks like this: 

\[jmx_hadoop*\]
env.category jvm
env.username monitor
env.password monitor
env.JRE_HOME /usr/lib/jvm/java-6-sun/jre
env.jmxurl service:jmx:rmi:///jndi/rmi://localhost:2103/jmxrm


Useful commands: 

/etc/init.d/munin-node restart 
chkconfig munin-node on
/usr/bin/munin-cron

Remove windows characters to file 

sed -i 's/\r//' /usr/share/munin/plugins/jmx_

Check logs: 

cd /var/log/munin/ 
cd /var/www/html/munin/ 
check munin-{var/log/update}.loge}.log

Thanks a lot to Stuart Teasdale for his precious help on installation and configuration!

You will find a link to jmx_ here And to jmxquery.jar here

Keren Ouaknine

Retrieved from "https://www.cs.huji.ac.il:80/wikis/MediaWiki/lawa/index.php?title=Munin_for_Hadoop&oldid=90"