Difference between revisions of "Building Hadoop CDH3"

Revision as of 18:40, 16 May 2011

Get The Source

• Cloudera Distribution of Hadoop (CDH3) - Download CDH3 from Cloudera. The source can be found in a Tarball format.

Software Prerequisites

The following software must be installed:

• Maven - on Ubuntu this can be achieved by running:

sudo apt-get install maven2

• zlib - on Ubuntu this can be achieved by running:

sudo apt-get install zlib1g zlib1g-dev

• You need g++. on Ubuntu this can be achieved by running:

sudo apt-get install g++

• Java 5 and 6.
• You need Forrest, download a release from here
• You need Xerces-C 2.8.x
  • Download it from here (3.1.x WILL NOT WORK!)
  • Build/Install instructions here
  • You can also download a binary distribution of Xerces-C.

Building

In order to build, run the following (after updating the variables with locations relevant for your machine):

export JAVA_6_HOME_32=/usr/lib/jvm/java-6-sun
export JAVA_5_HOME_32=/usr/lib/jvm/java-5-sun
export CFLAGS=-m32
export CXXFLAGS=-m32

export JAVA_6_HOME=${JAVA_6_HOME_32}
export JAVA_5_HOME=${JAVA_5_HOME_32}
export XERCES_HOME=/home/cantor/hadoop-workspace/hadoop-libs/xerces-c_2_8_0-x86-linux-gcc_3_4/xerces-c
export ECLIPSE_HOME=/usr/lib/eclipse
export FORREST_HOME=/home/cantor/hadoop-workspace/hadoop-libs/apache-forrest-0.8
ant -Dversion=cdh3.dev0 -Dcompile.native=true -Dcompile.c++=true -Dlibhdfs=1 -Dlibrecordio=true -Dxercescroot=${XERCES_HOME} -Declipse.home=${ECLIPSE_HOME} 
-Dforrest.home=${FORREST_HOME} -Djava5.home=${JAVA_5_HOME} clean api-report tar test test-c++-libhdfs

The bash script is available here: https://www.cs.huji.ac.il/project/lawa/downloadables/cdh3-buildscript.sh