MinneSPEC: A New SPEC Benchmark Workload for Simulation-Based Computer Architecture Research
The MinneSPEC inputs can be obtained by sending your SPEC license number to the SPEC webmaster at [email protected]. Upon confirming your license, the webmaster will send you a userid and password, which you can use to access the SPEC ftp site to obtain the MinneSPEC files.
Computer architects must determine how to most effectively use finite computational resources when running simulations to evaluate new architectural ideas. To facilitate efficient simulations with a range of benchmark programs, we have developed the MinneSPEC input set for the SPEC CPU 2000 benchmark suite. This new workload allows computer architects to obtain simulation results in a reasonable time using existing simulators. While the MinneSPEC workload is derived from the standard SPEC CPU 2000 workload, it is a valid benchmark suite in and of itself for simulation-based research. MinneSPEC also may be used to run large numbers of simulations to find ``sweet spots'' in the evaluation parameter space. This small number of promising design points subsequently may be investigated in more detail with the full SPEC reference workload. In the process of developing the MinneSPEC datasets, we quantify its differences in terms of function-level execution patterns, instruction mixes, and memory behaviors compared to the SPEC programs when executed with the reference inputs. We find that for some programs, the MinneSPEC profiles match the SPEC reference dataset program behavior very closely. For other programs, however, the MinneSPEC inputs produce significantly different program behavior. The MinneSPEC workload has been recognized by SPEC and is distributed with Version 1.2 and higher of the SPEC CPU 2000 benchmark suite.
These benchmarks are maintained by AJ KleinOsowski in the Department of Electrical and Computer Engineering at the University of Minnesota. Comments, questions, or problems can be directed to [email protected]
For more information, see: "MinneSPEC: A New SPEC Benchmark Workload for Simulation-Based Computer Architecture Research '', AJ KleinOsowski and David J. Lilja, Computer Architecture Letters, Volume 1, June, 2002. (A small correction, stated below, for this paper is available as: Laboratory for Advanced Research in Computing Technology and Compilers Technical Report No. ARCTiC 02-08, October, 2002). If you use the MinneSPEC inputs in your work, please cite the Computer Architecture Letters paper in any resulting publications. Thank you.
The April, 2003, release includes reduced input sets and profiles for all of the benchmarks in the SPEC CPU 2000 suite. Profiles are available on this site by clicking on the links for each benchmark. Input sets can be obtained by sending a request to [email protected]. In your request, be sure to include your SPEC CPU 2000 license number.
Note #1: The reference run of several SPEC CPU 2000 benchmarks use multiple runs of the same executable with different command line arguments or different input files. For these programs, we treat each of these different command lines as separate sub-benchmarks.
Note #2: The profiles below were generated by compiling the benchmarks for either the SimpleScalar PISA instruction set or the Alpha instruction set, then collecting profile information with the SimpleScalar suite of simulators. The Simulation ISA column in the table below documents which instruction set architecture was used for our profiles.
Note #3: The Input Reduction column documents what methods were used to develop the MinneSPEC workloads. The FuncProf and InstMix columns document where the MinneSPEC function-level execution profile and the instruction mix are nearly identical to the SPEC reference workload, as determined by the chi-squared test at O3 optimization for pisa ISA profiles, or O4 optimization for alpha ISA profiles. For programs with [YN], the MinneSPEC profile is compared to the multiple SPEC reference workloads for that program. The Run Length column documents the dynamic instruction count of each program, at O0 optimization.
Erratum to "MinneSPEC: A New SPEC Benchmark Workload for Simulation-Based Computer Architecture Research," AJ KleinOsowski and David J. Lilja, Computer Architecture Letters, Volume 1, June 2002.
It was recently discovered that, in the sim-profile simulator from the SimpleScalar version 3.0 suite, the counters used to track instruction mixes are explicitly declared to be unsigned ints, which makes them only 32 bits. As a result, when simulating programs with more than 4 billion (2**32) instructions, these counters overflow and produce incorrect outputs.
Since the benchmark programs in the SPEC CPU 2000 benchmark suite execute more than 4 billion (2**32) instructions with SPEC's standard reference input sets, simulating these programs with sim-profile causes the counts of the different types of instructions executed to overflow. Consequently, the instruction mix profile results discussed in this paper and posted on this web site prior to July, 2002, may contain errors.
The authors have modified sim-profile to allow it to correctly profile programs with more than 4 billion instructions and have collected corrected instruction mix profiles. These updated profiles are available below.
This error with the instruction mix profiles does not affect the use or distribution of the MinneSPEC workload. Only the instruction mix profiles, which are provided primarily as a convenience to MinneSPEC users, are affected. The other reported results, including the cache behavior gathered with sim-cache and the execution time profiles gathered with gprof, are unaffected by this counter overflow problem.
Furthermore, please note that the limited size of the counters in the standard distribution of sim-profile will affect all sim-profile users who run simulations longer than 4 billion instructions. If you fall into this category, we suggest that you download the stats.h and stats.c files. These two files can directly replace the existing stats.h and stats.c files in the SimpleScalar version 3.0 distribution. These files change the 32 bit sim-profile counters to 64 bit counters.
Many thanks to Hans Vandierendonck and Lieven Eeckhout of Ghent University for their detailed evaluation of the MinneSPEC workload which led to the discovery of these errors.
| Benchmark | Type | Description | Input Reduction Method | FuncProf | InstMix | Run Length | Profiles | Simulation ISA |
|---|---|---|---|---|---|---|---|---|
| 164.gzip,graphic | Int | Compression | modified command line, new input file | Y | Y | 2.5 billion | ref.graphic | pisa |
| 164.gzip,log | Int | Compression | modified command line, truncated ref file | N | Y | 1.0 billion | ref.log | pisa |
| 164.gzip,program | Int | Compression | modified command line, new input file | Y | Y | 4.4 billion | ref.program | pisa |
| 164.gzip,random | Int | Compression | modified command line, modified ref file | Y | Y | 1.9 billion | ref.random | pisa |
| 164.gzip,source | Int | Compression | modified command line, truncated ref file | N | Y | 2.4 billion | ref.source | pisa |
| 168.wupwise | Float | Quantum Chromodynamics | modified test file | N | Y | 14.6 billion | ref | alpha |
| 171.swim | Float | Shallow Water Modeling | same as test | N | Y | 2.8 billion | ref | alpha |
| 172.mgrid | Float | Muli-grid Solver | modified test file | N | Y | 993 million | ref | alpha |
| 173.applu | Float | Partial Differential Equations | modified test file | Y | Y | 952 million | ref | alpha |
| 175.vpr,place | Int | FPGA Place Route | same as test | Y | Y | 3.3 billion | ref.place | pisa |
| 175.vpr,route | Int | FPGA Place Route | same as test | Y | Y | 2.0 billion | ref.route | pisa |
| 176.gcc | Int | C Compiler | same as train | 5Y | 5Y | 6.4 billion | ref.166 , ref.200 , ref.expr , ref.integrate , ref.scilab | pisa |
| 177.mesa | Float | 3-D Graphics | modified command line, modified test file | N | N | 1.3 billion | ref | pisa |
| 178.galgel | Float | Computational Fluid Dynamics | modified test file | Y | N | 1.5 billion | ref | alpha |
| 179.art | Float | Image Recognition | modified command line | 2N | 2Y | 7.7 billion | ref.110 , ref.470 | pisa |
| 181.mcf | Float | Combinatorial Optimization | modified ref file | N | Y | 1.7 billion | ref | pisa |
| 183.equake | Float | Seismic Wave Propagation | modified train file | N | N | 1.7 billion | ref | pisa |
| 186.crafty | Int | Chess Playing | modified test file | Y | Y | 1.2 billion | ref | alpha |
| 187.facerec | Float | Image Processing | modified test file | N | N | 859 million | ref | alpha |
| 188.ammp | Float | Chemistry | truncated ref file | N | N | 2.2 billion | ref | pisa |
| 189.lucas | Float | Number Theory | modified test file | N | Y | 1.4 billion | ref | alpha |
| 191.fma3d | Float | Crash Simulation | modified test file | N | N | 774 million | ref | alpha |
| 197.parser | Int | Word Processing | sampled ref file | N | Y | 5.6 billion | ref | pisa |
| 200.sixtrack | Float | Nuclear Physics Accelerator Design | modified test file | N | N | 4.1 billion | ref | alpha |
| 252.eon,cook | Int | Computer Visualization | modified test file | Y | Y | 2.9 billion | ref | alpha |
| 252.eon,kajiya | Int | Computer Visualization | modified test file | Y | Y | 3.6 billion | ref | alpha |
| 252.eon,rushmeier | Int | Computer Visualization | modified test file | Y | Y | 3.1 billion | ref | alpha |
| 253.perlbmk | Int | PERL Language | same as ref.makerand | 1Y5N | 2Y4N | 1.8 billion | ref.535 , ref.704 , ref.850 , ref.957 , ref.makerand , ref.perfect | pisa |
| 254.gap | Int | Group Theory | modified test file | Y | Y | 1.0 billion | ref | alpha |
| 255.vortex | Int | Database | modified train file | 3Y | 3Y | 1.5 billion | ref.1 , ref.2 , ref.3 | pisa |
| 256.bzip2,graphic | Int | Compression | modified command line, modified ref file | Y | Y | 6.4 billion | ref.graphic | pisa |
| 256.bzip2,program | Int | Compression | modified command line, new input file | Y | Y | 5.0 billion | ref.program | pisa |
| 256.bzip2,source | Int | Compression | modified command line, sampled ref file | Y | Y | 3.8 billion | ref.source | pisa |
| 300.twolf | Int | Place and Route | sampled train file | Y | Y | 1.5 billion | ref | pisa |
| 301.apsi | Float | Pollutant Distribution | modified test file | Y | N | 1.8 billion | ref | alpha |
For information about this project from SPEC, visit http://www.spec.org/cpu2000/research/umn.
This work was supported in part by the National Science Foundation under grants EIA-9971666 and CCR-9900605, by the Minnesota Supercomputing Institute, and by Compaq's Alpha Development Group.