Google has announced that it will be making a MapReduce framework open-source to allow users to run native C and C++ code in their Hadoop environments. Hadoop can suffer from performance issues due to being written in Java so this framework will come as welcome news.
‘MR4C’ (MapReduce for C) was developed at Skybox Imaging for large-scale satellite image processing and geospatial data science use. Google found Hadoop a powerful solution for scalable data handling thanks to its job tracking and cluster management capabilities but wanted to be able to leverage the robust image-processing libraries written in C and C++.
Companies have built their own proprietary systems to achieve this, but MR4C offers a comprehensive solution to save time in working with large datasets. The framework is built around a few simple concepts which Google explains in its blog post:
Algorithms are stored in native shared objects that access data from the local filesystem or any uniform resource identifier (URI), while input/output datasets, runtime parameters, and any external libraries are configured using JavaScript Object Notation (JSON) files. Splitting mappers and allocating resources can be configured with Hadoop YARN based tools or at the cluster level for MRv1.
Workflows of multiple algorithms can be strung together using an automatically generated configuration. There are callbacks in place for logging and progress reporting which you can view using the Hadoop JobTracker interface. Your workflow can be built and tested on a local machine using exactly the same interface employed on the target cluster.
MR4C isn’t the first time Google has opted to use native C++ code for Hadoop. The Quantcast File System (in-house alternative for Hadoop’s Distributed File System) they use is also written in C++ due to its performance benefits. Facebook uses a similar ideology with their ‘HipHop’ system which converts SQL to C++ before executing it.
Apache Spark, a data-processing framework faster than MapReduce, is seeing a lot of interest but does not support C/C++ natively. It does however support Scale, Python, and Java, so it will be interesting to see which framework gains the most traction.
You can get started with MR4C via Google’s documentation and source code at the MR4C GitHub page.
Do you think MR4C or Apache Spark will gain the most traction for data-processing? Let us know in the comments.
