Overview of Hadoop Ecosystem

Of late, have been looking into the Big Data space and Hadoop in particular.  When I started looking into it, found that there are so many products and tools related to Haddop.   Using this post summarize my discovery about Hadoop Ecosystem.

Hadoop Ecosystem

A small overview on each is listed below:

Data Collection  - Primary objective of these is to move data into a Hadoop cluster

Flume: - Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store. Developed by cloudera and currently being incubated at Apache software foundaton. The details about the same can be found here.

Scribe: Scribe is a server for aggregating streaming log data. It is designed to scale to a very large number of nodes and be robust to network and node failures. Dveloped by Facebook and can be found here. 

Chuckwa: Chukwa is a Hadoop subproject devoted to large-scale log collection and analysis. Chukwa is built on top of the Hadoop distributed filesystem (HDFS) and MapReduce framework and inherits Hadoop’s scalability and robustness. Details about Chhuckwa can be found here. 

Kafka: Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. Kafka is being incubated at Apache software foundation and details about Kafka can be found here.

Sqoop -   Sqoop allows easy import and export of data from structured data stores such as relational databases, enterprise data warehouses, and NoSQL systems. Using Sqoop, you can provision the data from external system on to HDFS, and populate tables in Hive and HBase. cloudera is the creator of Sqoop and Sqoop is currently undergoing incubation at Apache Software Foundation. More information on this project can be found here.  

HIHO - HIHO aims to integrate Hadoop with existing data centric systems. There is a need to connect Hadoop to different systems, like databases, report display tools etc to fully leverage the functionality offered by each. HIHO project attempts to do so. Details on HIHO can be found here. 

Core Engine - These form the core Engine for storage and processing

HDFS - HDFS (Hadoop Distributed File System) is the primary distributed storage used by Hadoop applications. A HDFS cluster primarily consists of a NameNode that manages the file system metadata and DataNodes that store the actual data. HDFS is part of core Hadoop framework and is a subproject of Apache Hadoop project. Additonal information pertaining to HDFS can be found here. 

Map Reduce - Hadoop MapReduce is a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes. Good starting point to understand mapreduce would be here. 

Data Storage  - Specialized storage on Hadoop

HBase - HBase is the Hadoop database. We can think of it as a distributed, scalable, big data store. The most common usage scenario for HBase is to use it when you need random, realtime read/write access to your Big Data. HBase is a sub project of Apache Hadoop project and information on HBase can be found  here. 

Support Extensions - These help in extending and supporting core activities on Hadoop

Avro - Avro is an additon to apache family of products and address the area of Data serialization.  Apache Avro is a data serialization system that provides Rich data structures, A compact, fast, binary data format, A container file, to store persistent data, Remote procedure call (RPC) and integration with dynamic languages. Information on Avro can be found here. 

Zookeeper - ZooKeeper is part of Apache family of products and  is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications like Hadoop. Details about zookeeper can be found here. 

Oozie - Oozie is a workflow/coordination system to manage Apache Hadoop job. Oozie is being incubated at Apache software foundation and  details regarding oozie can be found here.  

Thrift - he Apache Thrift software framework, for scalable cross-language services development, combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, OCaml and Delphi and other languages. Thrift also provides a set of API to connect with HBase. Details on Thrift can be found here. 

Scripting Languages -  These provide easy ways to facilitate processing on Hadoop

Jaql - Jaql is a high-level scripting language for the JavaScript Object Notation (JSON). It is able to run on Hadoop and break most requests down to Map/Reduce tasks. Jaql heavily borrowes from SQL, XQuery, LISP, Pig Latin, JavaScript and Unix Pipes. Developed primarily inside Ibm and part of BigInsights project. The good starting point for JAQL will be here. 

Pig - Pig is a high-level scripting language for data transformation. It is a Hadoop Subproject and can be looked at as a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. Details about Pig can be found here. 

Hive - Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems. Hive is a subproject of Apache hadoop project and information w.r.t to Hive can be found here. 

Data Analytics   -  These are specialized software to perform Analytics on data stored on Hadoop

Intellicus : Intellicus, with a variety of powerful features, extends Ad hoc Reporting and Dash boarding solution to support Big Data. Intellicus provides data source connectivity through Hive to all variants of Hadoop like Apache and Cloudera. It provides both background batch processing and online query processing for Ad hoc reporting using HiveQL. Intellecus as a basic free edition but also has feature rich paid editions. The website for intellicus is https://www.intellicus.com

Karmasphere - Karmasphere provides self-service access to Big Data and analytic functions for faster, more efficient analysis and collaboration. Visually explore data for patterns and trends, iteratively analyze data using familiar queries and skills. Look up details of karmasphhere at  https://karmasphere.com

Monitoring and management - These help to monitor and manage the jobs and data on Hadoop

Gangalia: Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids.  It uses carefully engineered data structures and algorithms to achieve very low per-node overheads and high concurrency. Ganglia is a BSD-licensed open-source project that grew out of the University of California, Berkeley Millennium Project.  The details of Ganglia can be found here. 

Hue: Hue is both a web UI for Hadoop and a framework to create interactive web applications. It features a FileBrowser for accessing HDFS, JobSub and JobBrowser applications for submitting and viewing MapReduce jobs, a Beeswax application for interacting with Hive. Hue grew out of Cloudra and details pertaining to Hue can be found here. 

Cacti: Cacti is a complete network graphing solution designed to harness the power of RRDTool's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box.There is a good set of hadoop cacti templates that can be used to visualize the data exposed by Hadoop jmx. Look for details on cacti here. 

Hadoop high level application - These are application/frameworks which leverage Hadoop for specialized purposes

Search Related - These are software which leverage Hadoop in web crawling and indexing.

Nutch: Apache Nutch is an open source web-search software project. Nutch is a project of the Apache Software Foundation. Nutch is a Web crawler written in Java. By using it, we can find Web page hyperlinks in an automated manner. Nutch can run hadoop clusters and a good source to begin reading on this would be here. 

Solr: Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search.  Solr integrates easily with Nutch. Nutch crawls the web and solr indexes the crawled data.  Details of Solr can be found here. 

Machine Learning -  Specialized software that implement machine learning algorithms over Hadoop.

Mahout -  Mahout is a subproject in Apache family of products on Hadoop.  The Apache Mahout library's goal is to build scalable machine learning libraries. Mahout core algorithms for clustering, classfication and batch based collaborative filtering are implemented on top of Apache Hadoop using the map/reduce paradigm. A starting point for information on Mahout would be here.