Compiler and Architecture Seminar, November 11th, 2003 Subsetting SPEC When Measuring Results: Research vs. Industry A D’oh… Torus [email protected] IBM Labs in Haifa © 2003 IBM Corporation IBM Labs in Haifa Production History ISCA 2003 Panel, June 9th, 2003, San Diego, CA : Subsetting SPEC when measuring results: valid or manipulative? Mis ulation: Partial and Misleading Use of SPEC CPU 2000 in Computer Architecture Conferences IEEE Micro, July/August 2003 (Vol. 23, No. 4) The Use and Abuse of SPEC: An ISCA Panel Revenge of the SPEC 2/11 Subsetting SPEC When Measuring Results: Research vs. Industry © 2003 IBM Corporation IBM Labs in Haifa # of papers Research: Computer Architecture Conferences ISCA Micro HPCA ISCA Micro HPCA-8 Number of papers published: Papers that used a version of SPEC: Earliest conference deadline: SPEC CPU2000 announced: 3/11 Subsetting SPEC When Measuring Results: Research vs. Industry HPCA ISCA HPCA-9 209 140 (66%) December 2000 December 1999 © 2003 IBM Corporation IBM Labs in Haifa Industry: SPEC CPU2000 # of papers Quotes from www.spec.org “SPEC CPU2000 is the next-generation industry-standardized CPU-intensive benchmark suite.” “These benchmarks measure the performance of the processor, memory and compiler on the tested system.” Datapath 4/11 Memory Multi-Thread Media Simulation Subsetting SPEC When Measuring Results: Research vs. Industry Compiler Misc. © 2003 IBM Corporation IBM Labs in Haifa Full Use # of benchmarks used per paper # of benchmarks used per paperNo Reason - ( - ) - ( - ) ( ) Reason - ( - ) - ( - ) ( Given ) % percent of papers percent of papers Research: Partial Use of CPU2000 % % % % % % % Proc.% Proc. CINT CFP % Mem. CINT Mem. CFP Comp. CINT Comp. CFP # of Papers Proc. CINT Proc. CFP Mem. CINT Mem. CFP Comp. CINT Comp. CFP Authors omit benchmarks, suites, and explanations! 5/11 Subsetting SPEC When Measuring Results: Research vs. Industry © 2003 IBM Corporation IBM Labs in Haifa C Research: The The Chosen Tower ofFew Babel F77 F90 # of paper appearances # of paper appearances art equake ammp swim mesa applu C++ gzip vpr parser gcc mcf vortex mgrid lucas wupwise apsi galgel sixtrack facerec fma d twolf bzip perlbmk crafty gap eon Average of 8 (CINT) and 6 (CFP) Benchmarks per Paper! 6/11 Subsetting SPEC When Measuring Results: Research vs. Industry © 2003 IBM Corporation IBM Labs in Haifa Research vs. Industry: Top 10 Systems (CINT) Top Industry SPECint ratios ( / ) 7/11 Top Research SPECint ratios ( / ) Intel Pentium Intel Pentium AMD Athlon AMD Athlon AMD Opteron HP Itanium Dell Xeon SGI Itanium HP Itanium AMD Opteron HP Xeon Dell Xeon Fujitsu Pentium HP Xeon IBM Opteron Dell Pentium Einux Opteron IBM Opteron IBM POWER + Bull Itanium Subsetting SPEC When Measuring Results: Research vs. Industry © 2003 IBM Corporation IBM Labs in Haifa Research vs. Industry: CFP2000 105 papers used CINT2000, 63 used CFP2000 (60%) CFP2000 considered more regular and predictable CFP2000 has higher Dcache miss rate Only 35% of Memory-Hierarchy papers use a majority of CFP2000 496 CINT2000 and 488 CFP2000 results published by SPEC Linear Regression model of SPECratio by the CPU clock rate Multiple Linear Regression model of SPECratio by speed and MemoryHierarchy . % . % . . % PowerPC Alpha . % . % 8/11 . CINT CFP . . CFP CINT BP BP CFP CINT I$ I$ CFP CINT D$ D$ Linear Regression Multiple LR by MHz by MHz and MH Subsetting SPEC When Measuring Results: Research vs. Industry © 2003 IBM Corporation IBM Labs in Haifa Research vs. Industry: Top 10 Systems (CFP) Top Industry SPECfp ratios ( / ) HP Itanium HP Itanium SGI Itanium SGI Itanium Bull Itanium Bull Itanium Dell Itanium ION Itanium ION Itanium Dell Itanium IBM Power + Sun UltraSparc III Intel Pentium Fujitsu SPARC HP Alpha HP Alpha AMD Athlon IBM POWER + HP Alpha 9/11 Top Research SPECfp ratios ( / ) C Subsetting SPEC When Measuring Results: Research vs. Industry V NEC Itanium © 2003 IBM Corporation IBM Labs in Haifa Research vs. Industry: Processor Architecture MIPS Alpha Media/SIMD HP-PA IAPowerPC SPARC AMD x % % % % % % Research SPEC CINT CINT Research SPEC CFP CFP Use Architectures of the Future, not the Past! SimpleScalar – A Two-Edged Sword 10/11 Subsetting SPEC When Measuring Results: Research vs. Industry © 2003 IBM Corporation IBM Labs in Haifa Research Industry SPEC: Support research with simulation sized input sets Vendors: Get simulation tools out to the field 11/11 Subsetting SPEC When Measuring Results: Research vs. Industry © 2003 IBM Corporation