Main Page

Welcome to the LAWA Wiki

This site is dedicated to the arcane art of Hadooping (very) large data.

What is LAWA?

Please refer to: https://www.cs.huji.ac.il/wikis/MediaWiki/lawa/index.php/LawaWiki:Community_Portal

Getting started

• Configuring a Hadoop pseudo-cluster with one node (For development purposes).
• Configuring a Hadoop cluster.

Building Hadoop

• Building Hadoop 0.21.x from source.
• Building Cloudera's Hadoop Distribution (CDH3) from source

The Hadoop 0.21 Innards

• Hadoop Startup scripts de-mystified.

Hadoop HUJI Modifications

• Hadoop HUJI: Measurement hook-points.
• Hadoop HUJI: Scheduler hook-points

The Hadoop 0.21 Test Cluster

The test cluster for Hadoop 0.21 development is now live and its NameNode and JobTracker (master node) is located on hadoop23@cs. If you wish to access the web applications for the NameNode and JobTracker, please follow these links from inside the CS network:

NameNode statistics:

http://hadoop23:50070

JobTracker statistics:

http://hadoop23:50030

Processing of ARCs

The ArcTool package is now processing arc files. I uploaded the Jar with sources on /cs/phd/ouaknine/process_arc.jar Job run example: bin/hadoop jar ~/process_arc.jar ArcProcessing -libjars=/cs/phd/ouaknine/PhD/lawa/arcTools/heritrix-1.14.4.jar,/cs/phd/ouaknine/PhD/lawa/arcTools/fastutil-6.1.0.jar keren1GB/ /user/ouaknine/output/sum4