Hbase is the Hadoop database, allowing to manage your data in a table way more than in a file way.

Pig is a scripting language that will generate on the fly map reduce job to get the data you need. It is very compact compared to hand writing map reduce job.

One of the nice thing between Pig and Hbase is that they can be integrated. Thanks to recent patch committed.

The documentation is not well updated yet (currently almost relate to the patch itself) some can be found on some post like here but they all lack of details explanation. Even the Cloudera distribution CDH3 indicates support for this integration but no sample can be found.

Below I describe the installation and configuration steps to make the integration works, provide and example and finally expose some of the limits of the current release (0.8)

1. First, install the map reduce components (Job tracker and Task tracker). One Job tracker and many task tracker as you have data nodes. Each distribution may provide different procedure for the installation, I’m using the Cloudera CDH3 distrib, which for the map reduce installation is well documented.
2. Now proceed with the Pig installation,  it is also easy as long you are not trying the integration with Hbase.  You need only to install pig on the client side, you do not need to install it on each Data Node neither on the Name Node, but just on the machine where you want to run the pig program.
3. Check your installation by entering the the grunt shell (just enter ‘pig’ from the shell).
4. Now the tricky part – In order to use Pig/Hbase integration you in fact need to make Map Reduce jobs aware of Hbase classes, otherwise you will have “ClassNotFoundException” or worst the zookeeper exception like “org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase” during execution. The way to perform this easily without coping the hbase configurations into your hadoop configuration dir, is by using hadoop-env.sh and hbase to print its own classpath.
   So add to your hadoop-env.sh file file the following
   

   #define the location of hbase
   export HBASE_HOME=/usr/lib/hbase
   #Customize the classpath to make map reduce job aware of Hbase
   export HADOOP_CLASSPATH="`${HBASE_HOME}/bin/hbase classpath`:$HADOOP_CLASSPATH"
   </span>

5. You will also need pig to be aware of Hbase configuration, for this you can use the HBASE_CONF_DIR environment variable (for CDH release), which is configured by default to be /etc/hbase/conf,

Ok your installation should be fine now, so let’s do an example…. For this example let assume we have stored in HBase a schema named TestTable, and column family named A, we have also several fields named field0, field1,…, and we want to extract this information and store it into ‘results/extract’.  In this case the pig script will looks like:

my_data = LOAD 'hbase://TestTable' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('A:field0 A:field1', '-loadKey=true') as (id, field0, field1);

store my_data into 'results/extract' using PigStorage(';');

So the above script indicate that the my_data relation will contains the fields “field0, field1″ and the ID (due to the -loadKey parameter). These fields will be stored as id, field0, field1 under the ‘result/extract’ folder and values will be separated by semicolon.

You can also use some comparison operator on the key. The current operator supported are lt, lte, gt, gte for lower than, lower than or equal, greater than and greather than or equal.

 Note: There is no support for logical operator, you can use more than one comparison operator which are chained as AND.

Limitations:

The current HBaseStorage, does not allow the usage of wildcard, that is if you need all the fields in a row, you need to enumerate them. Wildcard are supported in version 0.9.

You can use HBaseStorage to store back the records in HBase nevertheless the HBase usage is incosistent a bug was already opened on this.

Below are some of my impression for this day

• Moore Law seems to reach limit – or at least it is not sure that the race to double the number of transistor each 18 months still the primary goal of the CMOS industry (too much core we barely need so much)
• Parallel programming, is one of the major topic that should be learn, and we should stop creating the “its complicated” barrier on the topic
• Security on the cloud data seems to have a major attention both by Microsoft and Intel
• Theory (Mathematics) and Hardware knowledge are important to achieve performant programming.

Resources

• Paxos Made Simple (Leslie Lamport)
• TCE – Technion Computer Engineering Conference
• The CMOS Crisis (Adjunct Prof. Doug Burger, Microsoft)

In this project a JAVA generator generates millions of lines of C++ code which of course have to be compiled, and if you are a C++ guy you certainly already have your hair standing on your head because of the time it will require to compile such huge amount of code. Well you are right, we faced extremely long compilation time (12+ hours in Unix, in windows…), which are major problem in a product that should have a quick time to market.

Worst, the product is used both under Windows and Unix platforms, which means that a solution need to be created for both worlds.

Under Windows – even with the usage of Incredibuild from Xoreax (a great grid compiler platform which allowed to reduce considerably the compilation time), the user still needed to wait 2 hours for compilation, which was not acceptable.

Under Unix – no grid compiler (unless you work only on few platform), we tried to use distcc but the results were still not satisfying and you need additional hardware. We where stuck…

Therefore we begun a research for an alternative that could speed up the builds, and for that we needed to understand the factor that affected compilation time, our main suspicious factor was the number of lines of code to compile, since the code was generated, it was very easy to inflate the output nevertheless we soon understood that we were wrong…

Below are factors impacting the compilation time (according to their impact)

1. In the first place the number of files to compile – this is one of the major factor affecting compilation time, the compiler is not really smart at reusing information it processed between invocation and is not able either to work on a set of files, and it is especially slow (I/O bound) for building dependencies. If you want to really reduce the compilation time reduce the number of file to compile, and it does not mean to write all in a single file, you can use what is called Unity Build.
   A Unity Build group several cpp files in a single one using just the include directive. For example let say you want to compile file1.cpp to file 10.cpp then create a new file group.cpp as follow:
   

   #include "file1.cpp"
   #include "file2.cpp"
   #include "file3.cpp"
   ..
   ..
   #include "file10.cpp"
   

   Now compile group.cpp and don’t forget to add the file1.cpp…file10.cpp location as include path.
   This method produces miracle (of course you have to balance the number of files you put in a single group/unity). Our compilation reduced from 15 hours to just 2.

2. Include paths – large number of include path directly affect the build time, since the compiler (or pre-compiler) need to scan all the path until it find the requested include. So try to minimize them or at least organize the path list according to the most searched one.
3. NAS (Network Attached Storage) also has a bad impact on the compilation (write is usually fast, but read is slow so library creation is slow).
4. Generate cppdep and compile on the same time – unix compilers support option to create cppdep file and compile at the same time you can save approx 20% of your compilation time.
5. Forward declaration, also know as the “Pimpl idiom” to reduce dependency, greatly help, the problem is that you cannot always refactor the code, to avoid some include that will erase your effort.
6. Usage of template – using C++ template excessively increase compilation time and libraries volume (especially if the template is declared in header).
7. Number of strings constants in a single file. It might should strange, but some compilers (HP and Sun at least) have a performance degradation when the compilation unit contains an large number of string (few thousand).
   Note: Visual Studio compiler is not sensitive to this factor.
   
8. Generic vs inflated code – using call to function or writing the content of function where you need them (like a forced “inline”). Inlining function in this way may produce better performance, but does not affect compilation time, as much as you think it affect. Effectively we reduced millions of lines of code by 75% using call to function instead of inlining their content, but we got no improvement in build time, but at least you gain more maintainable/debugable code.
9. Usage of pre-compiled header might help, but from our test they did not, the compilation time was in fact increased.
10. Usage of header cache folder – similarly to pre-compiled headers, should help (according to vendors) but from our test they most of the time do not.

So if you really want to reduce your compilation time try Unity Build concept, you will gain in:

1. Faster build time
2. Smaller objects size
3. Smaller libraries size
4. Better optimized code

Note: The compilation time is related to the number of cpp files in a single Unity Build (and their dependencies), and this number should be tuned according to the included file content (inline, template usage, headers used…). If you have too much file in a single unity/group file compilation time increase back (still better than when no using group file), nevertheless the library size declines (even if the compilation time increased back).

Mikael.Sitruk

With a jar file you don’t need scripts or long command line to setup your classpath for execution.  Nevertheless if you can do better than configuring the classpath and the main from command line, you can use the manifest file for this. Doing so, you can just type java -jar <command line args>

The manifest is a text file (property like) containing information on the archive, as part of this information you can define the main class of the archive and define the classpath (as long you did not pack other jar too)

In order to do so, define in the manifest the following tag ‘Class-Path’ and ‘Main-Class’. Following is a sample:

Main-Class: sample.package.MyMain
Class-Path: directory-one/sub-directory-one/referenced.jar directory-two/

Keep in mind that:

1. You specify several directories and/or referenced jar using a space as delimiter
2. Reference to directories and other jars are relative to the jar
3. Any referenced jar using the Class-Path attribute cannot be present in your original archive (without special classloader)
4. If you have resource in some directory don’t forget the slash at the end otherwise the content of the directory is not seen.

Resources

JAR specification

Ant is a powerful build and script tool provided by Apache Foundation. In a recent project I used the exec task and needed to allow usage of spaces in command line argument of the called executable. A legitimate request, but if you don’t pay attention on the different ’exec’  parameters syntax, you may waste a lot of time…

Handling space in “exec” task’s arguments.

The exec task of ant allow to execute system command. The arguments of the command are passed as arg sub tags. The sub tag is the arg tag, followed by an attribute named value, line or path. If your argument contains space (for example a file path) do not use the line attribute, instead use the value or path ones. The line attribute will consider the spaces as separator of command line argument for the program executed.

For example you need to pass a text file as command line to a document editing application, and that the file is located under C:\Documents and settings\msthoughts\doc.txt

Using the line attribute:

<exec executable="myapp">
  <arg line="c:\Documents and settings\msthoughts\doc.txt"/>
</exec>

Will produce an error since the command line is interpreted as 3 args each one denoted with brakets: [c:\Documents] [and] [settings\msthoughts\doc.txt]

To resolve this in command line window you usually use the quotes to wraps the command line arg having space. But using quote from ant will lead to other problem very quickly. The better solution is to use the value attribute instead of the line attribute

<exec executable="myapp">
  <arg value="c:\Documents and settings\msthoughts\doc.txt"/>
</exec>