Benchmarking for AC on the Cloud

Papers

Closely Related

Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance

• Run a variety of benchmarks (CPU, Memory, Disk, Network) on a large number of small and large EC2 instances over the course of a month
• Results are stratified corresponding to the two CPU types: Xeon and Opertron
• Xeon backed instances perform about twice as well as Opertron instances
• Distribution of processor types varies by availability zone

EC2 Performance Analysis for Resource Provisioning of Service-Oriented Applications

• Over a long time (hours) individual instances are stable in performance
• Over a short time (minutes) instances are can have sharp dips in performance
• Between instances of the same type, average performance can vary by a factor of 4

Exploiting Hardware Heterogeneity within the Same Instance Type of Amazon EC2

• Measures the distribution of processor types, both Xeon and Opertron and broken down into specific models
• Benchmarks the performance of each processor type
• Checks CPU information located in '/proc/cpuinfo'; the VM hypervisor does not modify this
• Outlines simple cost analysis of seeking out better performing instances

More for Your Money: Exploiting Performance Heterogeneity in Public Clouds

• Examines EC2 variation on three levels: inter-architecture, intra-architecture, and temporal
• measures significant variation in each case
• proposes both black-box(measured performance) and grey-box(based on knowledge of processor distributions) placement methods
• evaluates placement methods in simulations and on EC2

Somewhat Related

How is the Weather tomorrow? Towards a Benchmark for the Cloud

• Discusses cloud services in general and what they offer
• Lists important metrics to consider in cloud computing and how to go about writing a cloud-wide benchmark to measure these

Benchmarking in the Cloud: What it Should, Can, and Cannot Be

• Provides a general overview of benchmarks and what makes a good one
• Lists challenges to consider when developing a benchmark for the cloud; focuses on benchmarking entire cloud rather than individual nodes

A Performance Analysis of EC2 Cloud Computing Services for Scientific Computing

• Evaluates the EC2 cloud through a series of benchmarks to determine its suitability to scientific computing
• Concluded that its performance and reliability were low compared to a dedicated cluster and thus less desirable

Resource Provisioning of Web Applications in Heterogeneous Clouds

• Measures heterogeneity of EC2 instances in terms of CPU and memory performance and proposes provisioning instances to tasks that would best suit instances
• Focus on web services and maintaining the Service Level Objective

An Evaluation of Amazon's Grid Computing Services: EC2, S3 and SQS

• Evaluated EC2 and related services, seeming to focus on data transfers between EC2 and S3
• S3 delivers up to 5 times better performance to EC2 than outside locations, but its performance can vary significantly in all cases

Exploring the Performance Fluctuations of HPC Workloads on Clouds

• Measures the variability in runtimes of running various solvers on EC2 and futureGrid
• Runtime fluctuation increases when more cores are used for a solver

-- Main.geschd - 05 Aug 2013