Benchmarking < BETA

---+ Benchmarking for AC on the Cloud
---++ Papers
---+++ Closely Related
[[https://cs.uwaterloo.ca/~kdaudjee/courses/cs848/papers/schad10.pdf][Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance]]
   * Run a variety of benchmarks (CPU, Memory, Disk, Network) on a large number of small and large EC2 instances over the course of a month
   * Results are stratified corresponding to the two CPU types: Xeon and Opertron
   * Xeon backed instances perform about twice as well as Opertron instances
   * Distribution of processor types varies by availability zone
[[http://coolfire.insomnia247.nl/cgc/Paper2.pdf][EC2 Performance Analysis for Resource Provisioning of Service-Oriented Applications]]
   * Over a long time (hours) individual instances are stable in performance
   * Over a short time (minutes) instances are can have sharp dips in performance
   * Between instances of the same type, average performance can vary by a factor of 4
[[https://www.usenix.org/system/files/conference/hotcloud12/hotcloud12-final40.pdf][Exploiting Hardware Heterogeneity within the Same Instance Type of Amazon EC2]]
   * Measures the distribution of processor types, both Xeon and Opertron and broken down into specific models
   * Benchmarks the performance of each processor type
   * Checks CPU information located in '/proc/cpuinfo'; the VM hypervisor does not modify this
   * Outlines simple cost analysis of seeking out better performing instances
[[http://blogs.rsa.com/wp-content/uploads/gamingpaper.pdf][More for Your Money: Exploiting Performance Heterogeneity in Public Clouds]]
   * Examines EC2 variation on three levels: inter-architecture, intra-architecture, and temporal
   * measures significant variation in each case
   * proposes both black-box(measured performance) and grey-box(based on knowledge of processor distributions) placement methods
   * evaluates placement methods in simulations and on EC2
---+++ Somewhat Related
[[http://cs.brown.edu/~kraskat/pub/dbtest09-cloudbench.pdf][How is the Weather tomorrow? Towards a Benchmark for the Cloud]]
   * Discusses cloud services in general and what they offer
   * Lists important metrics to consider in cloud computing and how to go about writing a cloud-wide benchmark to measure these
[[http://www.pds.ewi.tudelft.nl/~iosup/cloud-benchmarking-what12tpctc-vldb.pdf][Benchmarking in the Cloud: What it Should, Can, and Cannot Be]]
   * Provides a general overview of benchmarks and what makes a good one
   * Lists challenges to consider when developing a benchmark for the cloud; focuses on benchmarking entire cloud rather than individual nodes
[[http://www.st.ewi.tudelft.nl/~iosup/ec2perf-sci-comp09cloudcomp.pdf][A Performance Analysis of EC2 Cloud Computing Services for Scientific Computing]]
   * Evaluates the EC2 cloud through a series of benchmarks to determine its suitability to scientific computing
   * Concluded that its performance and reliability were low compared to a dedicated cluster and thus less desirable
[[https://www.usenix.org/legacy/event/webapps11/tech/final_files/Dejun.pdf][Resource Provisioning of Web Applications in Heterogeneous Clouds]]
   * test
[[http://simson.net/clips/academic/2007.Harvard.S3.pdf][An Evaluation of Amazon's Grid Computing Services: EC2, S3 and SQS]]
   * test
[[http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5708474][Exploring the Performance Fluctuations of HPC Workloads on Clouds]]
   * test

-- Main.geschd - 05 Aug 2013
This topic: BETA > TipsAndTricks > WebHome > EmpiricalAlgorithmics > Benchmarking
Topic revision: r4 - 2013-08-07 - geschd