Hadoop Linux教学课件.ppt

Cryptography Hadoop 500G Google: 20亿 分布式文件系统 3:1 7:1 9:1 3:1 3:1 7:1 3:1 3：4 7：2 9：1 30亿，500亿 Map reduce: 函数式编程API 运行框架 1G 20T Slot: Cloud computing 海量数据? 大数据 Lucene Nutch(MapRedue+NDFS) Hadoop MapReduce HDFS Hadoop Map mapper: map task Reduce reducer: reduce task shuffle and sort combiner ? partitioner 100 word 1 a 1 how 1 word 1 20TB map A.txt 1GB:64MB Hadoop MapReduce 编程框架 运行环境 HDFS HDFS flume (ASF) chukwa (ASF) scribe (facebook) Hadoop MapReduce HDFS HBase RDB 行式数据库 BigTable, HBase (Data source, Data sink) 列式数据库 Functional programming MapReduce combiner HDFS HDFS Read HDFS Write Hadoop R,RHadoop Cloudera, CDH NoSQL: 稀疏格式存储方案 CRUD: 0.20.2 JobTracker TaskTracker NameNode SecondaryNameNode DataNode 本地模式 伪分布式模式 完全分布工模式 /hadoop/temp Hadoop data ingress and egress Key elements of ingress and egress idempotent aggregation data format transformation recoverability correctness resource consumption and performance monitoring Moving data into Hadoop Two primary methods that can be used for moving data into Hadoop writing external data at the HDFS level (a data push) reading external data at the MapReduce level (more like a pull). Data sources Hadoop data ingress across a spectrum of data sources log files semistructured or binary files Databases HBase Pushing log files into Hadoop Flume, Chukwa, and Scribe are log collecting and distribution frameworks that have the capability to use HDFS as a data sink for that log data FLUME A distributed system for collecting streaming data CHUKWA An Apache subproject of Hadoop that also offers a large-scale mechanism to collect and store data in HDFS SCRIBE A rudimentary streaming log distribution service, developed and used heavily by Facebook Pushing and pulling semistructured and binary files HDFS File Slurper A simple utility that supports copying files from a local directory into HDFS and vice versa Scheduling regular ingress activities with Oozie Oozie is a server based Workflow Engine specialized in running workflow