Apache Hadoop Data-Processing Framework

From GM-RKB
(Redirected from Apache Hadoop's HDFS)
Jump to navigation Jump to search

An Apache Hadoop Data-Processing Framework is a Java-based distributed fault-tolerant batch data-processing software framework.



References

2014

2013

  • http://en.wikipedia.org/wiki/Hadoop
    • Apache Hadoop is an open-source software framework that supports data-intensive distributed applications, licensed under the Apache v2 license. It supports the running of applications on large clusters of commodity hardware. The Hadoop framework transparently provides both reliability and data motion to applications. Hadoop implements a computational paradigm named MapReduce[1], where the application is divided into many small fragments of work, each of which may be executed or re-executed on any node in the cluster. In addition, it provides a distributed file system that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both map/reduce and the distributed file system are designed so that node failures are automatically handled by the framework.[2] It enables applications to work with thousands of computation-independent computers and petabytes of data. Hadoop was derived from Google's MapReduce and Google File System (GFS) papers.

      The entire Apache Hadoop “platform” is now commonly considered to consist of the Hadoop kernel, MapReduce and Hadoop Distributed File System (HDFS), as well as a number of related projects – including Apache Hive, Apache HBase, and others.[3]

      Hadoop is written in the Java programming language and is a top-level Apache project being built and used by a global community of contributors.[4] Hadoop and its related projects (Hive, HBase, Zookeeper, and so on) have many contributors from across the ecosystem.[5]

2012

2011

  • http://hadoop.apache.org/#What+Is+Hadoop%3F
    • QUOTE: The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.

      The project includes these subprojects:

      • Hadoop Common: The common utilities that support the other Hadoop subprojects.
      • Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
      • Hadoop MapReduce: A software framework for distributed processing of large data sets on compute clusters.


2010

  • http://wiki.apache.org/hadoop
    • QUOTE: Apache Hadoop is a framework for running applications on large cluster built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which may be executed or reexecuted on any node in the cluster. In addition, it provides a distributed file system (HDFS) that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both Map/Reduce and the distributed file system are designed so that node failures are automatically handled by the framework.


  • http://wiki.apache.org/hadoop/ProjectDescription
    • QUOTE: Map/Reduce is a programming paradigm that expresses a large distributed computation as a sequence of distributed operations on data sets of key/value pairs. The Hadoop Map/Reduce framework harnesses a cluster of machines and executes user defined Map/Reduce jobs across the nodes in the cluster. A Map/Reduce computation has two phases, a map phase and a reduce phase. The input to the computation is a data set of key/value pairs. …

      … The Hadoop Map/Reduce framework has a master/slave architecture. It has a single master server or jobtracker and several slave servers or tasktrackers, one per node in the cluster. The jobtracker is the point of interaction between users and the framework. Users submit map/reduce jobs to the jobtracker, which puts them in a queue of pending jobs and executes them on a first-come/first-served basis. The jobtracker manages the assignment of map and reduce tasks to the tasktrackers. The tasktrackers execute tasks upon instruction from the jobtracker and also handle data motion between the map and reduce phases.