- 1、本文档共22页,可阅读全部内容。
- 2、原创力文档(book118)网站文档一经付费(服务费),不意味着购买了该文档的版权,仅供个人/单位学习、研究之用,不得用于商业用途,未经授权,严禁复制、发行、汇编、翻译或者网络传播等,侵权必究。
- 3、本站所有内容均由合作方或网友上传,本站不对文档的完整性、权威性及其观点立场正确性做任何保证或承诺!文档内容仅供研究参考,付费前请自行鉴别。如您付费,意味着您自己接受本站规则且自行承担风险,本站不退款、不进行额外附加服务;查看《如何避免下载的几个坑》。如果您已付费下载过本站文档,您可以点击 这里二次下载。
- 4、如文档侵犯商业秘密、侵犯著作权、侵犯人身权等,请点击“版权申诉”(推荐),也可以打举报电话:400-050-0827(电话支持时间:9:00-18:30)。
查看更多
下一代分布式计算模型
© Hortonworks Inc. 2013
Page 1
YARN
Apache Hadoop Next Generation
Compute Platform
Bikas Saha
@bikassaha
© Hortonworks Inc. 2013 - Confidential
Apache Hadoop YARN
• Apache Hadoop
–De facto Big Data open source platform
–Running for about 5 years in production at hundreds of companies
like Yahoo, Ebay and Facebook
• Hadoop 2
–Significant improvements in HDFS distributed storage layer. High
Availability, NFS, Snapshots
–YARN – next generation compute framework for Hadoop designed
from the ground up based on experience gained from Hadoop 1
–YARN running in production at Yahoo for about a year
–YARN awarded Best Paper at SOCC 2013
Page 2
© Hortonworks Inc. 2013 - Confidential
Single App
BATCH
HDFS
Single App
INTERACTIVE
Single App
BATCH
HDFS
1st Generation Hadoop: Batch Focus
HADOOP 1.0
Built for Web-Scale Batch Apps
All other usage patterns
MUST leverage same
infrastructure
Forces Creation of Silos to
Manage Mixed Workloads
Single App
BATCH
HDFS
Single App
ONLINE
Page 3
© Hortonworks Inc. 2013 - Confidential
Hadoop 1 Architecture
JobTracker
Manage Cluster Resources Job Scheduling
TaskTracker
Per-node agent
Manage Tasks
Page 4
© Hortonworks Inc. 2013 - Confidential
Hadoop 1 Limitations
Lacks Support for Alternate Paradigms and Services
Force everything needs to look like Map Reduce
Iterative applications in MapReduce are 10x slower
Scalability
Max Cluster size ~5,000 nodes
Max concurrent tasks ~40,000
Availability
Failure Kills Queued Running Jobs
Hard partition of resources into map and reduce slots
Non-optimal Resource Utilization
Page 5
Our Vision: Hadoop as Next-Gen Platform
MapReduce
(cluster resource management
data processing)
HDFS
(redundant,reliable storage)
© Hortonworks Inc. 2013 - Confidential
YARN
(cluster resource management)
HDFS2
(redundant,highly-available reliable storage)
MapReduce
(data processing)
Others
Single Use System
Batch Apps
HADOOP 1.0
Multi Purpose Platform
Batch, Interactive, Onli
文档评论(0)