Required Software and Dependencies
--------------------------------------
gcc and g++
Thrift RPC Library (developer version)
Intel's Thread Building Blocks (TBB) Library (developer version)
Boost Libraries (developer versions)
libfkfsClient -- Kosmos File System + Dependencies
Building libhdfs
--------------------------------------
1) Building libhdfs
# Reference URL:
# http://www.mail-archive.com/common-user@hadoop.apache.org/msg01812.html
# This line compiles it!
JAVA_HOME=/usr/lib/jvm/default-java ant compile -Dcompile.c++=true -Dlibhdfs=true
2) Running HDFS
a) Format filesystem: bin/hadoop namenode -format
b) Run DFS: bin/start-dfs.sh
Building Minni
--------------------------------------
*** Note: Full system builds have only been completed under Ubuntu 10.04 LTS.
1) For git build run:
./bootstrap.sh
./configure --with-kfs=/home/wolf/Desktop/kosmostest/kfs-0.5/build/
make
2) For tarball build run:
./configure --with-kfs=/home/wolf/Desktop/kosmostest/kfs-0.5/build/
make
Running Minni
--------------------------------------
Start up HDFS, then all worker nodes. Then start master/MapReduce application.
cd master/ && make test
That will cause a test to run of the MapReduce system (things should be
mostly built already).
Notes
--------------------------------------
[General]
+ Only supported DFS is KFS currently. Both HDFS and KFS is targeted for future support.
+ Would like to have XML logging interface for easy instrumentation.
[Master]
+ The master has some Thrift RPC code, but currently it only acts as
a client to a hard-coded list of Worker nodes. There is no uninitiated
communication from nodes to the master (all polling-based communication).
+ The master is currently single-threaded (not depending on TBB) because
that is simple for now (ease of development and debugging), and it wasn't necessary yet.
+ src/master/example/Driver.cpp shows a sample MapReduce application
using our current style (not very abstract yet).
[Worker Node]
STUB
[Client API]
STUB
