| |
| META TOPICPARENT |
name="EmpiricalAlgorithmics" |
Benchmarking for AC on the Cloud |
| |
- the micro instance variation is consistent with Amazon's description: "Micro instances (t1.micro) provide a small amount of consistent CPU resources and allow you to increase CPU capacity in short bursts when additional cycles are available."
- Variations between instances of the same type is also small (COV < 6%). This contradicts what some papers claimed. However, looking at the underlying CPU types show very little variation
- CPUs mostly were Xeon E5-2650 with only a few Xeon E5645
|
|
<
< |
- the region/availability zone must be fairly homogenous
|
>
> |
- the region/availability zone must be fairly homogeneous
|
| | |
|
>
> | Spear Run Experiment
Another small experiment to test variability of performance. This one focused on testing variations over short periods of time by repeatedly running many small Spear runs and checking the stability of the runtimes.
- I tested four instance types: T1.micro, M1.small, M1.medium, M1.xlarge
- four instances of each type were tested
- each instance ran 80 problems from SW_verification/Wine through Spear
- each problem was tested 30 times consecutively to measure for runtime variations
- The median completion time for problems was 1.7s (wallclock) so it measured variations at a small scale
- Both wall clock and CPU times were taken
The experiment results: SpearResultsWallclock.xls , SpearResultsCPUTime.xls
- CoVs were calculated for all 80 problems on each instance as a measure of the performance variation over the course of the 30 runs
- wallclock and CPU time results were very similar
- Wallclock CoV (avg, max):
- t1.micro : 64.0%, 145.2%
- m1.small : 2.7%, 15.4%
- m1.medium : 1.0%, 11.7%
- m1.xlarge : 0.9%, 4.1%
- Micro instances are very unstable, which is to be expected
- other instance types seem to be fairly stable over time
- only measured over the course of a few hours in a single region. Possibly a more busy time/region would see worse effects
|
| |
Related Work
Closely Related |
|
<
< | Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance |
>
> | Runtime Measurements in the Cloud: Observing, Analysing, and Reducing Variance |
| |
- Run a variety of benchmarks (CPU, Memory, Disk, Network) on a large number of small and large EC2 instances over the course of a month
|
|
<
< |
- Results are stratified corresponding to the two CPU types: Xeon and Opertron
- Xeon backed instances perform about twice as well as Opertron instances
|
>
> |
- Results are stratified corresponding to the two CPU types: Xeon and Opteron
- Xeon backed instances perform about twice as well as Opteron instances
|
| |
- Distribution of processor types varies by availability zone
EC2 Performance Analysis for Resource Provisioning of Service-Oriented Applications
- Over a long time (hours) individual instances are stable in performance
- Over a short time (minutes) instances are can have sharp dips in performance
- Between instances of the same type, average performance can vary by a factor of 4
Exploiting Hardware Heterogeneity within the Same Instance Type of Amazon EC2 |
|
<
< |
- Measures the distribution of processor types, both Xeon and Opertron and broken down into specific models
|
>
> |
- Measures the distribution of processor types, both Xeon and Opteron and broken down into specific models
|
| |
- Benchmarks the performance of each processor type
- Checks CPU information located in '/proc/cpuinfo'; the VM hypervisor does not modify this
- Outlines simple cost analysis of seeking out better performing instances
|
| | -- Main.geschd - 05 Aug 2013
| META FILEATTACHMENT |
attr="h" comment="" date="1376095762" name="UnixBenchResults.xls" path="UnixBenchResults.xls" size="22016" user="geschd" version="1.1" |
|
|
>
> |
| META FILEATTACHMENT |
attr="h" comment="" date="1376434749" name="SpearResultsCPUTime.xls" path="SpearResultsCPUTime.xls" size="1013248" user="geschd" version="1.1" |
| META FILEATTACHMENT |
attr="h" comment="" date="1376434775" name="SpearResultsWallclock.xls" path="SpearResultsWallclock.xls" size="1034240" user="geschd" version="1.1" |
|
Copyright © 2008-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.