Abstract

Validation is an important part of the development process of real-time systems. During validation, it should be proved that, the system meets its timing constraints. Therefore, worst-case execution time (WCET) analysis is performed. Currently, several WCET analysis tools are being developed. Researchers who develop new WCET analysis tools, need benchmarks to evaluate and compare their tools with alternatives. These benchmarks are called WCET benchmarks. In this paper a new WCET benchmark suite named ABench2020 is introduced. Its main focus is to provide benchmark programs in Ada programming language for WCET research. Therefore, ABench2020 includes several benchmark programs which were written in Ada programming language. The benchmark programs implement different program structures and properties to help researchers test their systems from different aspects. ABench2020 was published as open source. It is freely available over the Internet.

References

[1] Buttazzo GC. Hard Real-Time Computing Systems. Boston, MA: Springer US; 2011. doi: 10.1007/978-1-4614-0676-1

[2] Kopetz H. Real-Time Systems. 2nd ed. Boston, MA: Springer US; 2011. doi: 10.1007/978-1-4419-8237-7

[3] Bollella G et al. The Real-Time Specification for Java. Addison-Wesley; 2000.

[4] Dobbing B, Burns A. The Ravenscar Tasking Profile for High Integrity Real-Time Programs. Proceedings of the 1998 annual ACM SIGAda international conference on Ada - SIGAda '98; 1998; 1-6. doi: 10.1145/289524.289525

[5] AdaCore. AdaCore homepage. 20-Jul-2020. [Online]. Available: https://www.adacore.com

[6] Wilhelm R et al. The worst-case execution-time problem—overview of methods and survey of tools. ACM Trans. Embed. Comput. Syst.; 2008; 7(3): 1–53. doi: 10.1145/1347375.1347389 

[7] Davis RI, Cucu-Grosjean L. A Survey of Probabilistic Timing Analysis Techniques for Real-Time Systems. Leibniz Trans. Embed. Syst.; 2019; 6(1): 03:1-03:60. doi: 10.4230/LITES-v006-i001-a003

[8] Cazorla FJ, Kosmidis L, Mezzetti E, Hernandez C, Abella J, Vardanega T. Probabilistic Worst-Case Timing Analysis: Taxonomy and Comprehensive Survey. ACM Comput. Surv.; 2019; 52(1): 14:1-14:35. doi: 10.1145/3301283

[9] Abella J et al. WCET analysis methods: Pitfalls and challenges on their trustworthiness. 10th IEEE International Symposium on Industrial Embedded Systems (SIES); 2015; 1–10. doi: 10.1109/SIES.2015.7185039

[10] Gustafsson J, Betts A, Ermedahl A, Lisper B. The Mälardalen WCET Benchmarks: Past, Present And Future. 10th International Workshop on Worst-Case Execution Time Analysis (WCET 2010); 2010; 136–146. doi: 10.4230/OASIcs.WCET.2010.136

[11] Sakarya University. Real-Time Systems Research Laboratory homepage. 20-Jul-2020. [Online]. Available: https://rtsrlab.sakarya.edu.tr

[12] Tian X et al. BigDataBench-S: An Open-Source Scientific Big Data Benchmark Suite. 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW); 2017; 1068–1077. doi: 10.1109/IPDPSW.2017.111

[13] Bienia C, Kumar S, Singh JP, Li K. The PARSEC Benchmark Suite: Characterization and Architectural Implications. Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques - PACT’08; 2008; 72-81. doi: 10.1145/1454115.1454128

[14] Che S et al. Rodinia: A benchmark suite for heterogeneous computing. 2009 IEEE International Symposium on Workload Characterization (IISWC); 2009; 44–54. doi: 10.1109/IISWC.2009.5306797

[15] Haynes B, Mazumdar A, Balazinska M, Ceze L, Cheung A. Visual Road: A Video Data Management Benchmark. Proceedings of the 2019 International Conference on Management of Data - SIGMOD’19; 2019; 972–987. doi: 10.1145/3299869.3324955

[16] Wang Y et al. Benchmarking OpenMP programs for real-time scheduling. 2017 IEEE 23rd International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA); 2017; 1–10. doi: 10.1109/RTCSA.2017.8046322

[17] Guthaus MR, Ringenberg JS, Ernst D, Austin TM, Mudge T, Brown RB. MiBench: A free, commercially representative embedded benchmark suite. Proceedings of the Fourth Annual IEEE International Workshop on Workload Characterization. WWC-4 (Cat. No.01EX538); 2001; 3–14. doi: 10.1109/WWC.2001.990739

[18] EEMBC. Embedded Microprocessor Benchmark Consortium homepage. 20-Jul-2020. [Online]. Available: https://www.eembc.org

[19] Standard Performance Evaluation Corporation. Spec - Standard Performance Evaluation Corporation. 20-Jul-2020. [Online]. Available: https://www.spec.org

[20] Blackburn SM et al. The DaCapo Benchmarks: Java Benchmarking Development and Analysis. Proceedings of the 21st Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages, and Applications - OOPSLA’06; 2006; 169-190. doi: 10.1145/1167473.1167488

[21] Kalibera T, Parizek P, Haddad G, Leavens GT, Vitek J. Challenge Benchmarks for Verification of Real-Time Programs. Proceedings of the 4th ACM SIGPLAN Workshop on Programming Languages Meets Program Verification – PLPV’10; 2010; 57-62. doi: 10.1145/1707790.1707800

[22] Schoeberl M, Preusser TB, Uhrig S. The Embedded Java Benchmark Suite JemBench. Proceedings of the 8th International Workshop on Java Technologies for Real-Time and Embedded Systems; 2010; 120–127. doi: 10.1145/1850771.1850789

[23] Serttaş S, Şahin VH. PBench: A Parallel, Real-Time Benchmark Suite. Academic Perspective Procedia; 2018; 1(1): 178–186. doi: 10.33793/acperpro.01.01.37

[24] Kuzhan M, Şahin VH. MBBench: A WCET Benchmark Suite. Sak. Univ. J. Comput. Inf. Sci.; 2020; 3(1): 39–49. doi: 10.35377/saucis.03.01.705777

[25] Falk H et al. TACLeBench: A Benchmark Collection to Support Worst-Case Execution Time Research. 16th International Workshop on Worst-Case Execution Time Analysis (WCET 2016); 2016; 55: 2:1–2:10. doi: 10.4230/OASIcs.WCET.2016.2

[26] Nemer F, Cassé H, Sainrat P, Bahsoun JP, Michiel MD. PapaBench: a Free Real-Time Benchmark. 6th International Workshop on Worst-Case Execution Time Analysis (WCET’06); 2006; 4: 1-6. doi: 10.4230/OASIcs.WCET.2006.678

[27] Kluge F, Rochange C, Ungerer T. EMSBench: Benchmark and Testbed for Reactive Real-Time Systems. Leibniz Trans. Embed. Syst.; 2017; 4(2): 02–1-02:23. doi: 10.4230/LITES-v004-i002-a002

[28] Wägemann P, Distler T, Hönig T, Sieh V, Schröder-Preikschat W. GenE: A Benchmark Generator for WCET Analysis. 15th International Workshop on Worst-Case Execution Time Analysis (WCET 2015); 2015; 47: 33–43. doi: 10.4230/OASIcs.WCET.2015.33

[29] Eichler C, Distler T, Ulbrich P, Wägemann P, Schröder-Preikschat W. TASKers: A Whole-System Generator for Benchmarking Real-Time-System Analyses. 18th International Workshop on Worst-Case Execution Time Analysis (WCET 2018); 2018; 63: 6:1-6:12. doi: 10.4230/OASIcs.WCET.2018.6