26 / 2018-02-14 14:01:03

A Self-tuning System for Big Data Analytics

Datanodes,data,HDFS,Mapping,

全体主题 > report all papers

Timely and cost-effective analytics over “Big Data” is now a key
ingredient for success in many businesses, scientific and engineering disciplines, and government endeavors. The Hadoop software
stack—which consists of an extensible MapReduce execution engine, pluggable distributed storage engines, and a range of procedural to declarative interfaces—is a popular choice for big data analytics. Most practitioners of big data analytics—like computational
scientists, systems researchers, and business analysts—lack the expertise to tune the system to get good performance. Unfortunately,
Hadoop’s performance out of the box leaves much to be desired,
leading to suboptimal use of resources, time, and money (in payas-you-go clouds). We introduce Starfish, a self-tuning system for
big data analytics. Starfish builds on Hadoop while adapting to user
needs and system workloads to provide good performance automatically, without any need for users to understand and manipulate the
many tuning knobs in Hadoop. While Starfish’s system architecture
is guided by work on self-tuning database systems, we discuss how
new analysis practices over big data pose new challenges; leading
us to different design choices in Starfish.Timely and cost-effective analytics over “Big Data” is now a key
ingredient for success in many businesses, scientific and engineering disciplines, and government endeavors. The Hadoop software
stack—which consists of an extensible MapReduce execution engine, pluggable distributed storage engines, and a range of procedural to declarative interfaces—is a popular choice for big data analytics. Most practitioners of big data analytics—like computational
scientists, systems researchers, and business analysts—lack the expertise to tune the system to get good performance. Unfortunately,
Hadoop’s performance out of the box leaves much to be desired,
leading to suboptimal use of resources, time, and money (in payas-you-go clouds). We introduce Starfish, a self-tuning system for
big data analytics. Starfish builds on Hadoop while adapting to user
needs and system workloads to provide good performance automatically, without any need for users to understand and manipulate the
many tuning knobs in Hadoop. While Starfish’s system architecture
is guided by work on self-tuning database systems, we discuss how
new analysis practices over big data pose new challenges; leading
us to different design choices in Starfish.