The 1st International Workshop on Benchmark Engineering for Software Engineering (BESE)

In software engineering (SE) research, high-quality program benchmarks play an essential role in evaluating novel approaches. However, SE community lacks proper benchmark engineering (BE) practices. Similar to software engineering, BE should have guidelines for benchmark requirement, design, development and verification. Currently, the existing repositories have unsystematically addressed only the last two items. However, without benchmark requirements and design descriptions, it is difficult for a researcher to assess the benchmark's adequacy for an evaluation. With the advent of frameworks that obtain programs from software repositories, the need of identifying systematic BE processes is especially urgent.

The one-day BESE workshop aims to initiate a dialog on benchmark engineering processes through invited talks, presentations and open discussions on the following topics: the current state of benchmark availability and quality for different SE domains; guidelines for designing high-quality benchmarks; important benchmark features; and improving the usability of benchmarks. BESE solicits short and extended abstract papers outlining the needs of benchmark users, insights and experience from benchmark designers and creators. The workshop's outcome is forming a community of BE researchers.

Important Dates

Submissions due:June 7, 2019
Author Notification:June 24, 2019
Camera-ready due:July 1, 2019