hadoop内核在虚拟化平台上的优化与扩展_5723.doc

Hadoop Virtualization Extensions Junping Du Sr.MTS, VMware, Inc ? 2009 VMware Inc. All rights reserved Cloud: Big Shifts in Simplification and Optimization 1. Reduce the Complexity 2. Dramatically Lower Costs 3. Enable Flexible, Agile IT Service Delivery to simplify operations and maintenance to redirect investment into value-add opportunities to meet and anticipate the needs of the business 2 A Unified Analytics Cloud Significantly Simplifies . Simplify ? Single Hardware Infrastructure ? Faster/Easier provisioning SQLCluster Big SQL NoSQL Hadoop NoSQL Cluster Unifed Analytics Infrastructure Private Public Hadoop Cluster . Optimize ? Shared Resources = higher utilization ? Elastic resources = faster on-demand access Decision Support Cluster 3 Unifying the Big Data Platform using Virtualization . Goals ? Make it fast and easy to provision new data clusters on demand ? Allow Mixing of Workloads ? Leverage virtual machines to provide isolation (esp. for Multi-tenant) ? Optimize data performance based on virtual topologies ? Make the system reliable based on virtual topologies . Leveraging Virtualization ? Elastic scale ? Use high-availability to protect key services, e.g., Hadoop’s namenode/job tracker ? Resource controls and sharing: re-use underutilized memory, cpu ? Prioritize Workloads: limit or guarantee resource usage in a mixed environment Cloud Infrastructure Private Public 4 VMware is committed to be the Best Virtual platform for Hadoop . Performance Studies and Best Practices ? Studies through 2010-2011 of Hadoop 0.20 on vSphere 5 ? White paper, including detailed configurations and recommendations . Making Hadoop run well on vSphere ? Performance optimizations in vSphere releases ? VMware engagement in Hadoop Community effort ? Supporting key partners with their distributions on vSphere ? Contributing enhancements to Hadoop ? Automate Hadoop deployment on vSphere . Hadoop Framework Integration Serengeti ? Spring for Hadoop: Enabling Spring to simplify