GAP logo

Benchmark
Suite

Scott Beamer   •   David Patterson   •   Krste Asanović

The unfortunate lack of a widely used graph benchmark suite forces each research publication to create its own evaluation methodology, and this often results in mistakes or unnecessary differences. Common serious mistakes we have observed include: using trivially small input graphs, using only a single input graph topology, or using low-performance implementations as baselines. These methodological issues make it difficult for good ideas to stand out and cloud the reasoning behind why these ideas are beneficial.

In order for the research community to make progress on accelerating graph processing, it is important to be able to properly and reliably compare results. We created the GAP Benchmark Suite to standardize evaluations in order to alleviate the methodological issues we observed. Through standardization, we hope to not only make results easier to compare, but to also prevent common evaluation mistakes. We provide both a benchmark specification to standardize the methodology and a high-performance reference implementation to be used as a baseline. Our benchmark was co-designed with our workload characterization, and it has undergone multiple revisions guided by community feedback.

Benchmark Specification

View on arXiv

To remove ambiguity, we specify:

Reference Code

View on GitHub

References

  1. The GAP Benchmark Suite, Scott Beamer, Krste Asanović, and David Patterson, arXiv:1508.03619 [cs.DC], 2015. arXiv
  2. "Locality Exists in Graph Processing: Workload Characterization on an Ivy Bridge Server", Scott Beamer, Krste Asanović, and David Patterson, International Symposium on Workload Characterization (IISWC), Atlanta, October 2015. PDF IEEE
    Best Paper Award
  3. "Understanding and Improving Graph Algorithm Performance", Scott Beamer, Ph.D. Thesis, University of California Berkeley, September 2016. PDF TR
    SPEC Kaivalya Dixit Distinguished Dissertation Award

External-User Publications

  1. "Whirlpool: Improving Dynamic Cache Management with Static Data Classification", Anurag Mukkara, Nathan Beckmann, and Daniel Sanchez, Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2016. ACM
  2. "Energy Efficient Architecture for Graph Analytics Accelerators", Mustafa Ozdal, Serif Yesil, Taemin Kim, Andrey Ayupov, John Greth, Steven M. Burns, Ozcan Ozturk, International Symposium on Computer Architecture (ISCA), 2016. ACM
  3. "Optimizing Indirect Memory References with milk", Vladimir Kiriansky, Yunming Zhang, and Saman Amarasinghe, International Conference on Parallel Architectures and Compilation (PACT), 2016. ACM

GAP Project Home