HBase examples on OS-X and Maven

Ok, so today I needed to get HBase 0.20.0 running on my local os-x box, simply in standalone mode. I am starting a project where I need to manage 50-100 million records and I wanted to try out HBase.

Here are the steps I took, the steps below are a consolidation of some pointers found in the HBase and Hadoop quick start guides.

A) Download HBase 0.20.X (currently 0.20.0), extract and install to /my/dir/hbase

B) Make sure your shell environment is setup to point to your Java 1.6 Home and your PATH is setup correctly which should be something like:

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home
export HBASE_HOME=/my/dir/hbase
PATH=$PATH:$HBASE_HOME/bin:$JAVA_HOME:bin
export PATH

C) Even though we are running in standalone mode. HBase is built on top of Hadoop and Hadoop uses SSH to communicate with masters/slaves. So we need to make sure the process can ssh to the localhost without a passphrase. (My standalone setup of HBase would not start properly without this).

Lets check to see if you can SSH locally without a password. Type ssh localhost. If this fails, we need to permit this so run the following two commands:

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh localhost (you should be able to connect now)

D) Ok, at this point you should be able to fire up HBase, lets do the following:

/my/dir/hbase/bin/start-hbase.sh

Once started up type: /my/dir/hbase/bin/hbase shell this brings up the interactive console, sort of like the mysql console where you can directly interact with the database or in this case, not a database, but the HBase KV store. While on the console type status 'detailed'. If you get successful output we are good to go! Hbase is running, type exit to get back to the bash shell. Lets leave HBase running

MAVEN INTEGRATION

Ok, now we need to setup your Java classpath to include the HBase jars. They are all located at /my/dir/hbase/lib. If you are using Maven and want to get HBase configured in your project. You can use the following UN-OFFICIAL HBase Maven POM and deploy script listed below. These files were originally provided by Fivecloud's post located here and I upgraded them for HBase 0.20.0.

Deploy Script For Maven Dependancies

#! /bin/sh
#
# Deploy all HBase dependencies which are not available via the official
#	 maven repository at http://repo1.maven.org.
#
#
# This is for HBase 0.20.0
#
# Modified for HBase 0.20.0 from the original located at
# http://www.fiveclouds.com/2009/04/13/deploying-hbase-to-your-local-maven-repo/
#
# The maven repository to deploy to.
#

REPOSITORY_URL=file:///$HOME/.m2/repository

if [ -z $HBASE_HOME ]; then
	echo "Error: HBASE_HOME is not set." 2>&1
	exit 1
fi

HBASE_LIBDIR=$HBASE_HOME/lib


# HBase
#
mvn deploy:deploy-file -Dfile=$HBASE_HOME/hbase-0.20.0.jar \
	-DpomFile=hbase.pom -Durl=$REPOSITORY_URL

#Hadoop
mvn deploy:deploy-file -DgroupId=org.apache -DartifactId=hadoop \
	-Dversion=0.20.0 -Dpackaging=jar -Durl=$REPOSITORY_URL \
	-Dfile=$HBASE_LIBDIR/hadoop-0.20.0-plus4681-core.jar

#thrift
mvn deploy:deploy-file -DgroupId=com.facebook -DartifactId=thrift \
	-Dversion=r771587 -Dpackaging=jar -Durl=$REPOSITORY_URL \
	-Dfile=$HBASE_LIBDIR/libthrift-r771587.jar

#apache commons cli
mvn deploy:deploy-file -DgroupId=commons-cli -DartifactId=commons-cli \
	-Dversion=2.0-SNAPSHOT -Dpackaging=jar -Durl=$REPOSITORY_URL \
	-Dfile=$HBASE_LIBDIR/commons-cli-2.0-SNAPSHOT.jar
	
#zookeeper
mvn deploy:deploy-file -DgroupId=org.apache.hadoop -DartifactId=zookeeper \
	-Dversion=r785019-hbase-1329 -Dpackaging=jar -Durl=$REPOSITORY_URL \
	-Dfile=$HBASE_LIBDIR/zookeeper-r785019-hbase-1329.jar


# EOF

Unofficial "hbase.pom"

<?xml version="1.0" encoding="UTF-8"?>

<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hbase</artifactId>
  <packaging>jar</packaging>
  <version>0.20.0</version>

  <name>Hadoop HBase</name>

  <dependencies>
  
    <dependency> 
      <groupId>org.apache.hadoop</groupId>
      <artifactId>zookeeper</artifactId>
      <version>r785019-hbase-1329</version>
    </dependency>
  
    <dependency>
      <groupId>commons-cli</groupId>
      <artifactId>commons-cli</artifactId>
      <version>2.0-SNAPSHOT</version>
    </dependency>
    <dependency>
      <groupId>commons-collections</groupId>
      <artifactId>commons-collections</artifactId>
      <version>3.2</version>
    </dependency>
    <dependency>
      <groupId>commons-httpclient</groupId>
      <artifactId>commons-httpclient</artifactId>
      <version>3.0.1</version>
    </dependency>
    <dependency>
      <groupId>commons-logging</groupId>
      <artifactId>commons-logging</artifactId>
      <version>1.1.1</version>
    </dependency>
    <dependency>
      <groupId>commons-math</groupId>
      <artifactId>commons-math</artifactId>
      <version>1.1</version>
    </dependency>

    <dependency>
      <groupId>org.apache</groupId>
      <artifactId>hadoop</artifactId>
      <version>0.20.0</version>
    </dependency>

    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>1.2.13</version>
    </dependency>

    <dependency>
      <groupId>jetty</groupId>
      <artifactId>org.mortbay.jetty</artifactId>
      <version>5.1.4</version>
    </dependency>
    <dependency>
      <groupId>jline</groupId>
      <artifactId>jline</artifactId>
      <version>0.9.91</version>
    </dependency>
    <dependency>
      <groupId>com.facebook</groupId>
      <artifactId>thrift</artifactId>
      <version>r771587</version>
    </dependency>
    <dependency>
      <groupId>org.apache.lucene</groupId>
      <artifactId>lucene-core</artifactId>
      <version>2.2.0</version>
    </dependency>
    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>1.2.15</version>
    </dependency>
    <dependency>
      <groupId>xmlenc</groupId>
      <artifactId>xmlenc</artifactId>
      <version>0.52</version>
    </dependency>
    <dependency>
	    <groupId>org.apache.geronimo.specs</groupId>
	    <artifactId>geronimo-j2ee_1.4_spec</artifactId>
	    <version>1.0</version>
	    <scope>provided</scope>
    </dependency>

  </dependencies>

	<repositories>
		<repository>
			<id>virolab.cyfronet.pl</id>
			<name>virolab.cyfronet.pl (used for commons-cli-2.0)</name>
			<url>http://virolab.cyfronet.pl/maven2</url>
		</repository>
	</repositories>

</project>

LETS ACTUALLY USE HBASE...

F) Now it's time to fire up a Java app to do some basic HBase operations. The examples below are simple tests that are setup to run in JUnit within Spring on my box, so you can ignore the method names and the @Test annotations, as the meat of the examples are in the method bodies.

CREATE A TABLE EXAMPLE

Ok, HBase is NOT an RDBMS but a Big Table implementation. Think of it as a giant Hashtable with more advanced features. However for the purposes of this example I will speak in terms of Rows/Columns etc which are similar in concept to that of a database and are what most folks are familiar with.

The MOST important thing is that you COPY the /my/dir/hbase/conf/*.xml default HBase configuration files to someplace on your classpath. These files can be customized, but for straight out of the box testing they work as-is. Just MAKE SURE they are on your classpath before starting as HBaseConfiguration instances look for them there.

HBaseConfiguration config = new HBaseConfiguration(); 
HBaseAdmin admin = null;
try {
	// HBaseAdmin is where all the "DDL" like operations take place in HBase
	admin = new HBaseAdmin(config);
} catch(MasterNotRunningException e) {
	throw new Exception("Could not setup HBaseAdmin as no master is running, did you start HBase?...");
}

if (!admin.tableExists("testTable")) {
	admin.createTable(new HTableDescriptor("testTable"));
	
	// disable so we can make changes to it
	admin.disableTable("testTable");
	
	// lets add 2 columns
	admin.addColumn("testTable", new HColumnDescriptor("firstName"));
	admin.addColumn("testTable", new HColumnDescriptor("lastName"));
	
	// enable the table for use
	admin.enableTable("testTable");

}

	
// get the table so we can use it in the next set of examples
HTable table = new HTable(config, "testTable");

After running the above code fire up the HBase shell /my/dir/hbase/bin/hbase shell and once up, type "list" at the hbase shell prompt, you should see your testTable listed! Yeah!

ADD A ROW TO THE TABLE

// lets put a new object with a unique "row" identifier, this is the key
// HBase stores everything in bytes so you need to convert string to bytes
Put row = new Put(Bytes.toBytes("myID"));

/* lets start adding data to this row. The first parameter
is the "familyName" which essentially is the column name, the second
parameter is the qualifier, think of it as a way to subqualify values
within a particular column. For now we won't so we just make the qualifier
name the same as the column name. The last parameter is the actual value
to store */

row.add(Bytes.toBytes("firstName"),Bytes.toBytes("firstName"),Bytes.toBytes("joe"));
row.add(Bytes.toBytes("lastName"),Bytes.toBytes("lastName"),Bytes.toBytes("smith"));

try {
	// add it!
	table.put(row);
} catch(Exception e) {
    // handle me!
}

Ok, now go back to the HBase shell and type count 'testTable' and you should get one record accounted for. Good to go!

GET A ROW FROM THE TABLE

// a GET fetches a row by it's identifier key
Get get = new Get(Bytes.toBytes("myID"));

Result result = null;
try {
	// exec the get
	 result = table.get(get);
} catch(Exception e) {
	// handle me
}

// not found??
if (result == null) {
	// NOT FOUND!
}

// Again the result speaks in terms of a familyName(column)
// and a qualifier, since ours our both the same, we pass the same
// value for both
byte[] firstName = Bytes.toBytes("firstName");
byte[] lastName = Bytes.toBytes("lastName");
byte[] fnameVal = result.getValue(firstName,firstName);
byte[] lnameVal = result.getValue(lastName,lastName);

System.out.println(new String(fnameVal) + " " + new String(lnameVal));

DELETE A ROW FROM THE TABLE

Delete d = new Delete(Bytes.toBytes("myID"));

try {
	table.delete(d);
} catch(Exception e) {
	// handle me
}

Now fire up the hbase shell again and type "count 'testTable'", you should now get zero rows.

OK, well I hope that helped you get up and running with some HBase basics!

Advertisements

5 comments

  1. alex

    Very interesting read! You gained a new follower to this blog 😉

    I have a quick question regarding your small experiment, hope it’s not too late 🙂

    Can you add anything about performance?
    Almost everywhere I look I get the feeling that this is still rather slow for a small to medium data set.

    A timed query would be very interesting: one row, vs those 50 to 100 million records you were talking about at the beginning of the article.

  2. bitsofinfo

    Hey Alex –
    Sorry but I don’t have any performance data yet. I wrote this article after creating an HBase implementation of an abstract “data store” but have been side tracked since I wrote this and I did not have time to tweak HBase to handle a ton of concurrent additions on a Hadoop cluster I was working with.

    I can tell you that HBase is not really intended to be used for small/medium data sets, but instead very, very large data sets. Depending on your RDBMS setup 50-100m records may not even be considered “very, very large”.

  3. ubhay

    Can you add information on how to compile my java code and how do i run it. I am a newbie here so please explain it in detail.

    Step F in your blog:

    LETS ACTUALLY USE HBASE…

    F) Now it’s time to fire up a Java app to do some basic HBase operations. The examples below are simple tests that are setup to run in JUnit within Spring on my box, so you can ignore the method names and the @Test annotations, as the meat of the examples are in the method bodie

    Also can you please attach the full code.

  4. Ray

    Great post! I finally see a post that gives me a good start on HBase.
    I have tons of data in mysql, any easy way to import those to HBase and give a shot?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s