Charting the Benchmarking Landscape – A Survey of Benchmarks for Modern Memory Architectures
DOI:
https://doi.org/10.66834/4gb7n382Keywords:
Memory Benchmarks, Literature Survey, Memory Devices, Storage DevicesAbstract
Benchmarks play a crucial role in evaluating system components, comparing algorithmic advancements, and assessing software implementations. While well‑established benchmark suites exist for various domains, modern memory technologies, such as high‑bandwidth memory and near‑memory computing, lack standardized benchmarking methodologies. Researchers often resort to custom benchmarks or adapt existing ones, making comparative analyses across different technologies challenging. This survey provides a structured overview of benchmarks used in modern memory architectures, catering to both newcomers and experienced researchers. It builds on top of a structured literature review of 1937 benchmark‑related publications obtained via Google Scholar in 2024, out of which 940 were within the scope of this survey, covering a total of 834 unique benchmarks. Out of these, we categorize and analyze 17 proposed and 739 existing benchmarks by manual analysis, identify trends in their adoption, and highlight gaps in the benchmarking landscape. By offering insights into benchmarking methodologies and their evolution, this survey aims to guide researchers in selecting appropriate benchmarks for memory system evaluations, fostering consistency and comparability in future research.
References
1. S. Li, D. Reddy, and B. Jacob, "A performance & power comparison of modern high-speed DRAM architectures," in Proceedings of the International Symposium on Memory Systems, ser. MEMSYS '18. New York, NY, USA: Association for Computing Machinery, 2018, pp. 341–353, doi:https://doi.org/10.1145/3240303.3240315. DOI: https://doi.org/10.1145/3240302.3240315
2. S. A. McKee, "Reflections on the Memory Wall," in Proceedings of the 1st Conference on Computing Frontiers, ser. CF '04. New York, NY, USA: Association for Computing Machinery, 2004, pp. 162–167, doi:https://doi.org/10.1145/977109.977115. DOI: https://doi.org/10.1145/977091.977115
3. T. M. Hollis et al., "Recent Evolution in the DRAM Interface: Mile-Markers Along Memory Lane," IEEE Solid-State Circuits Magazine, vol. 11, no. 2, pp. 14–30, 2019, doi:https://doi.org/10.1109/ISSCC.2019.8691617. DOI: https://doi.org/10.1109/MSSC.2019.2910617
4. Micron Technology, Inc. (2023) Product Brief – Micron HBM3E. Online: Available: https://www.micron.com/content/dam/micron.com/products/product-briefs/product-flyer/hbmse-product-brief.pdf
5. H. Huang et al., "Shuhai: A Tool for Benchmarking High Bandwidth Memory on FPGAs," IEEE Transactions on Computers, vol. 71, no. 5, pp. 1133–1144, 2022, doi:https://doi.org/10.1109/TC.2021.3075765. DOI: https://doi.org/10.1109/TC.2021.3075765
6. C. H. Lam, History of Phase Change Memories. Boston, MA, USA: Springer US, 2009, pp. 1–14, doi:https://doi.org/10.1007/978-0-387-84874-7_1. DOI: https://doi.org/10.1007/978-0-387-84874-7_1
7. A. M. Rudoff, (2016) Deprecating the PCOMMIT Instruction. Intel Corporation. Online: Available: https://www.intel.com/content/www/us/en/develop/articles/technical/deprecate-pcommit-instruction.html
8. T. Abell, (2023) Apr hard disk drive (HDD) unit shipments worldwide from 1st quarter 2016 to 2022. Statista. Online: Available: https://www.statista.com/statistics/38951/global-shipment-figures-for-hard-disks-drives/
9. Western Digital. (2025) Blue 2.5 inch PC Mobile Hard Drive. Western Digital. Online: Available: https://www.westerndigital.com/en-us/products/internal-drives/wd-blue-mobile-sata-hdd/#WD5000LPZX
10. ——. (2025) WD Red Pro 3.5in NAS Hard Drive. Western Digital. Online: Available: https://www.westerndigital.com/en-us/products/internal-drives/wd-red-pro-sata-hdd/#WD2000FSPX
11. Seagate. (2024) Transition to Advanced Format 4K Sector Hard Drives. Seagate. Online: Available: https://www.seagate.com/www/en-us/standards/4k-sector-hard-drives-master-t2/
12. Y. Shiroishi et al., "Future Options for HDD Storage," IEEE Transactions on Magnetics, vol. 45, no. 10, pp. 3816–3822, 2009, doi:https://doi.org/10.1109/TMAG.2009.2020879. DOI: https://doi.org/10.1109/TMAG.2009.2024879
13. L. Gough, (2019) Dec Hard Drive Performance Over the Years. Tom's Hardware. Online: Available: https://www.tomshardware.com/guides/general/the-hard-disk-drive-hard-drive-performance-over-the-years/
14. S. Nadel and V. Kamenov, "HDD performance research," in Proceedings of the International Conference on Computer Science, ser. CS '18. Berlin, Germany: ResearchGate, 2018, pp. 106–111.
15. F. Chen, D. A. Koufaty, and X. Zhang, "Understanding Intrinsic Characteristics and System Implications of Flash Memory based Solid State Drives," in Proceedings of the 11th International Joint Conference on Measurement and Modeling of Computer Systems, ser. SIGMETRICS '09. New York, NY, USA: Association for Computing Machinery, 2009, pp. 181–192, doi:https://doi.org/10.1145/1555349.1555371. DOI: https://doi.org/10.1145/1555349.1555371
16. Samsung. (2021) ssd 870 evo sata III 2.5in — 250 GB. Samsung. Online: Available: https://www.samsung.com/us/computing/memory-storage/solid-state-drives/ssd-870-evo-250gb--mz-77e250b-am/
17. ——. (2023) 990 PRO NVMe M.2 SSD — 4 TB. Samsung. Online: Available: https://www.samsung.com/us/computing/memory-storage/solid-state-drives/ssd-990-pro-4tb-nvme-mz-v9p4t0b-am/
18. L. Smith. (2021) Samsung 870 EVO SSD Review. Storage Review. Online: Available: https://www.storagereview.com/review/samsung-870-evo-ssd-review
19. S. Downing. (2023) Samsung 990 Pro 4TB Review: The Best Gets Bigger. Tom's Hardware. Online: Available: https://www.tomshardware.com/reviews/samsung-990-pro-4tb-ssd-review
20. S. Corda, G. Singh, A. A. Awan, R. Jordans, and H. Corporaal, "Platform Independent Software Analysis for Near Memory Computing," in Proceedings of the 22nd International EurPar Conference on Digital System Design, pp. 232–239. New York, NY, USA: IEEE, 2019, pp. 606–609, doi:https://doi.org/10.1109/DSD.2019.00093. DOI: https://doi.org/10.1109/DSD.2019.00093
21. D. Lee et al., "Optimizing Data Movement with Near-Memory Acceleration of In-Memory DBMS," in Proceedings of the 22nd International Conference on Extending Database Technology, ser. EDBT '20. Konstanz, Germany: OpenProceedings, 2020, pp. 371–374, doi:https://doi.org/10.5441/002/edbt.2020.35.
22. B. Friese, M. Luebke Dreimann, and O. Spinczyk, "Performance Models for Task-based Scheduling with Disruptive Memory Technologies," in Proceedings of the 2nd Workshop on Disruptive Memory Systems, ser. DIMES '24. New York, NY, USA: Association for Computing Machinery, 2024, pp. 1–8, doi:https://doi.org/10.1145/3698783.3699376. DOI: https://doi.org/10.1145/3698783.3699376
23. J. Gómez-Luna, I. El Hajj, I. Fernandez, C. Giannoulas, G. F. Oliveira, and O. Mutlu, "Benchmarking a new Paradigm: Experimental Analysis and Characterization of a Real Processing-in-Memory System," IEEE Access, vol. 10, pp. 52505–52608, 2022, doi:https://doi.org/10.1109/ACCESS.2022.3174101. DOI: https://doi.org/10.1109/ACCESS.2022.3174101
24. F. Ferdaus and M. T. Rahman, Security of Emerging Memory Chips. Cham: Springer International Publishing, 2021, pp. 357–390, doi:https://doi.org/10.1007/978-3-030-64448-2_14. DOI: https://doi.org/10.1007/978-3-030-64448-2_14
25. J. Yang, Y. Kim, M. Hoseinzadeh, J. Izraelevitz, and S. Swanson, "An Empirical Guide to the Behavior and Use of Scalable Persistent Memory," in Proceedings of the 18th USENIX Conference on File and Storage Technologies, ser. FAST '20. Santa Clara, CA: USENIX Association, Feb. 2020, pp. 169–182. Online: Available: https://www.usenix.org/conference/fast20/presentation/yang
26. R. Bharath and K. S. Pande, "HBM3 Architecture Specific Checks and Timing Closure," in Proceedings of the 5th International Conference on Smart Electronics and Communication (ICOSEC), ser. ICOSEC '24. New York, NY, USA: IEEE, 2024, pp. 239–244, doi:https://doi.org/10.1109/ICOSEC61287.2024.10722076. DOI: https://doi.org/10.1109/ICOSEC61587.2024.10722076
27. JC-42.3 Subcommittee on DRAM Memories, "JEDEC Standard 79-5: DDR5 SDRAM." JEDEC Solid State Technology Association, Standard, Apr. 2024.
28. K. Kim and M.-j. Park, "Present and Future Challenges of High Bandwidth Memory (HBM)," in Proceedings of the International Memory Workshop, ser. IMW '24. New York, NY, USA: IEEE, 2024, pp. 1–4, doi:https://doi.org/10.1109/IMW.2024.10536972. DOI: https://doi.org/10.1109/IMW59701.2024.10536972
29. R. Blasing et al., "Magnetic Racetrack Memory: From Physics to the Cusp of Applications Within a Decade," Proceedings of the IEEE, vol. 108, no. 8, pp. 1303–1321, 2020, doi:https://doi.org/10.1109/JPROC.2020.2975719. DOI: https://doi.org/10.1109/JPROC.2020.2975719
30. A. Fazio, "Advanced Technology and Systems of Cross Point Memory," in Proceedings of the International Electron Devices Meeting, ser. IEDM '20. New York, NY, USA: IEEE, 2020, pp. 24.1.1–24.1.4, doi:https://doi.org/10.1109/IEDM13553.2020.031976. DOI: https://doi.org/10.1109/IEDM13553.2020.9371976
31. Z. Song et al., "From Ordered structure motif to sub-threshold phase transitions in phase change materials for data storage," Science China Information Sciences, vol. 61, 2018, doi:https://doi.org/10.1007/s11432-017-9410-2. DOI: https://doi.org/10.1007/s11432-018-9404-2
32. Samsung. (2025) SSDs (Solid State Drive). Samsung. Online: Available: https://www.samsung.com/us/computing/memory-storage/solid-state-drives/
33. Kingston. (2024) Solid State Drives (SSDs) for laptops, desktop PCs and servers. Kingston Technology. Online: Available: https://www.kingston.com/en/ssd?cf_chl_1=dp-deep-dive-nvme
34. JC-42.3 Subcommittee on DRAM Memories, "JEDEC Standard 79-43: DDR4 SDRAM." JEDEC Solid State Technology Association, Standard, Sep. 2020.
35. ——. "JEDEC Standard 235D: High Bandwidth Memory (DRAM HBM3)." JEDEC Solid State Technology Association, Standard, Jan. 2023.
36. ——. "JEDEC Standard 235D: High Bandwidth Memory (DRAM HBM1, HBM2)." JEDEC Solid State Technology Association, Standard, Mar. 2021.
37. S. Bertolazzi et al., "Nonvolatile Memories Based on Graphene and Related 2D Materials," Advanced Materials, vol. 31, no. 10, p. 1806663, 2019, doi:https://doi.org/10.1002/adma.201806663. DOI: https://doi.org/10.1002/adma.201806663
38. J. J. Kan et al., "A Study on Practically Unlimited Endurance of STT-MRAM," IEEE Transactions on Electron Devices, vol. 64, no. 9, pp. 3639–3646, 2017, doi:https://doi.org/10.1109/TED.2017.2731959. DOI: https://doi.org/10.1109/TED.2017.2731959
39. S. Munira and N. Khan, "Advances in resistive switching based memory devices," Journal of Physics D: Applied Physics, vol. 52, no. 43, p. 433002, 2019, doi:https://doi.org/10.1088/1361-6463/ab2e9e. DOI: https://doi.org/10.1088/1361-6463/ab2e9e
40. S. Kargaon and F. Nawab, "Challenges and future directions for energy, latency, and lifetime improvements in NVMs," Distributed Parallel Databases, vol. 41, no. 3, pp. 163–189, Sep. 2022, doi:https://doi.org/10.1007/s10619-022-07421-x. DOI: https://doi.org/10.1007/s10619-022-07421-x
41. Intel Corporation. (2023) Intel Optane Persistent Memory 200 Series Brief. Intel Corporation. Online: Available: https://www.intel.com/content/www/us/en/products/docs/memory-storage/persistent-memory/optane-persistent-memory-200-series-brief.html
42. A. A. Isalbil, S. Mittal, M. A. Al-Bayati, and P. B. Sumra, "A survey of techniques for architecting SLC/MLC/FLC hybrid flash memory-based SSDs," Concurrency and Computation: Practice and Experience, vol. 30, no. 13, p. 4218, 2018, doi:https://doi.org/10.1002/cpe.4210. DOI: https://doi.org/10.1002/cpe.4420
43. S.-I. Jang, S.-K. Yoon, K. Park, G.-H. Park, and S.-D. Kim, "Data Classification Management with its Interfacing Structure for Hybrid SLC/MLC PRAM Main Memory," The Computer Journal, vol. 58, no. 11, pp. 2852–2863, 2015, doi:https://doi.org/10.1093/comjnl/bxu133. DOI: https://doi.org/10.1093/comjnl/bxu133
44. S. Luber. (2024) What is Racetrack Memory? Vogd IT-Media GmbH. Online: Available: https://www.storage-insider.de/was-ist-racetrack-memory-racetrack-speicher-a-8e8616d5077c329e21de1b3921fa/
45. A. van Renen, L. Vogel, V. Leis, T. Neumann, and A. Kemper, "Persistent Memory I/O Primitives," in Proceedings of the 15th International Workshop on Data Management on New Hardware, ser. DaMoN '19. New York, NY, USA: Association for Computing Machinery, 2019, doi:https://doi.org/10.1145/3331601.3331605. DOI: https://doi.org/10.1145/3329785.3329930
46. L. Benson, L. Pupke, and T. Rabl, "PerMA-Bench: benchmarking persistent memory access," in Proceedings of the VLDB Endowment, vol. 15, no. 11, pp. 2463–2476, Jul. 2022, doi:https://doi.org/10.14778/3533180.3531976. DOI: https://doi.org/10.14778/3551793.3551807
47. S. S. Parkin, "Shiftable magnetic shift register and method of using the same," dec. 2004, US Patent 6,834,005, Filed June 18, 2003, Issued Dec. 21, 2004.
48. D. Zhang, N. Jayasena, A. Lyashevsky, J. L. Greathouse, L. Xu, and M. Ignatowski, "TOP-PIM: throughput-oriented programmable processing in memory," in Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing, ser. HPDC '14. New York, NY, USA: Association for Computing Machinery, 2014, pp. 85–98, doi:https://doi.org/10.1145/2600212.2600213. DOI: https://doi.org/10.1145/2600212.2600213
49. F. Devaux, "The true Processing In Memory accelerator," in Proceedings of the 31st Hot Chips Symposium, ser. HCS '19. New York, NY, USA: IEEE, 2019, pp. 1–24, doi:https://doi.org/10.1109/HOTCHIPS.2019.8876580. DOI: https://doi.org/10.1109/HOTCHIPS.2019.8875680
50. B. Friesel, M. L. Liebmann Dreimann, and O. Spinczyk, "A Full-System Perspective on UPMEM Performance," in Proceedings of the 1st Workshop on Disruptive Memory Systems, ser. DIMES '23. New York, NY, USA: Association for Computing Machinery, 2023, pp. 1–7, doi:https://doi.org/10.1145/3609308.3625266. DOI: https://doi.org/10.1145/3609308.3625266
51. Y.-C. Kwon et al., "A 228Gb 8-High Function-In-Memory DRAM, Based on HBM2 with a 1.2Tbps Programmable Computing Unit Using Bank-Level Parallelism, for Machine Learning Applications," in Proceedings of the IEEE International Solid-State Circuits Conference, ser. ISSCC '21, vol. 64. New York, NY, USA: IEEE, 2021, pp. 350–352, doi:https://doi.org/10.1109/ISSCC42613.2021.9365862. DOI: https://doi.org/10.1109/ISSCC42613.2021.9365862
52. S. Lee et al., "Hardware Architecture and Software Stack for PIM Based on Commercial DRAM Technology," in Proceedings of the 48th Annual International Symposium on Computer Architecture, ser. ISCA '21. New York, NY, USA: IEEE, 2021, pp. 43–56, doi:https://doi.org/10.1109/ISCA52012.2021.00013. DOI: https://doi.org/10.1109/ISCA52012.2021.00013
53. P. Gao et al., "Technical Exploration of RRAM Crossbar Arrays for Matrix-Vector Multiplication," in Proceedings of the 20th Asia and South Pacific Design Automation Conference, ser. ASP-DAC '20. New York, NY, USA: IEEE, 2015, pp. 106–111, doi:https://doi.org/10.1109/ASPDAC.2015.7058899. DOI: https://doi.org/10.1109/ASPDAC.2015.7058989
54. A. Siemenik et al., "OCC: An Automated End-to-End Machine Learning Optimized Compiler for Computing-In-Memory," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 41, no. 6, pp. 1674–1686, 2022, doi:https://doi.org/10.1109/TCAD.2021.3101464. DOI: https://doi.org/10.1109/TCAD.2021.3101464
55. V. R. Datti and P. Sridevi, "Performance Evaluation of Content Addressable Memories," in Proceedings of the 7th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions), ser. ICRITO '18. New York, NY, USA: IEEE, 2018, pp. 596–598, doi:https://doi.org/10.1109/ICRITO.2018.8748808. DOI: https://doi.org/10.1109/ICRITO.2018.8748808
56. H. Farzaneh, J. P. C. De Lima, M. Li, A. A. Khan, X. Su, and J. Castrillón, "CISCAM: A Compiler for CAM-based In-memory Accelerators," in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ser. ASPLOS '24. New York, NY, USA: Association for Computing Machinery, 2024, pp. 164–177, doi:https://doi.org/10.1145/3620666.3651386. DOI: https://doi.org/10.1145/3620666.3651386
57. V. Seshadri et al., "Ambit: in-memory accelerator for bulk bitwise operations using commodity DRAM technology," in Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO '17. New York, NY, USA: Association for Computing Machinery, 2017, pp. 273–287, doi:https://doi.org/10.1145/3123939.3123441. DOI: https://doi.org/10.1145/3123939.3124544
58. H. Farzaneh et al., "SHERLOCK: Scheduling Efficient and Reliable Bulk Bitwise Operations in NVMs," in Proceedings of the 61st ACM/IEEE Design Automation Conference, ser. DAC '24. New York, NY, USA: Association for Computing Machinery, 2024, doi:https://doi.org/10.1145/3649329.3654885. DOI: https://doi.org/10.1145/3649329.3658485
59. M. Moreau et al., "ReRe: ReRAM-based Logic Operations for Computing in Memory," in Proceedings of the International Conference on Very Large Scale Integration, ser. VLSI-SoC '18. New York, NY, USA: IEEE, 2018, pp. 192–195, doi:https://doi.org/10.1109/VLSI-SoC.2018.8644780. DOI: https://doi.org/10.1109/VLSI-SoC.2018.8644780
60. D. S. Shar, R. Blankenship, and D. Bergman, "An Introduction to the Compute Express Link (CXL) Interconnect," ACM Computing Surveys, vol. 56, no. 11, Jul. 2024, doi:https://doi.org/10.1145/3603699. DOI: https://doi.org/10.1145/3669900
61. D. Gouk, M. Kwon, H. Bae, S. Lee, and M. Jung, "Memory Pooling With CXL," IEEE Micro, vol. 43, no. 2, pp. 48–57, 2023, doi:https://doi.org/10.1109/MM.2023.3237491. DOI: https://doi.org/10.1109/MM.2023.3237491
62. C. Guo, J. Xu, and M. Sukman, "Disaggregated Architectures and the Redesign of Data Center Ecosystems: Scheduling, Pooling, and Infrastructure Trade-Offs," IEEE Communications Magazine, vol. 64, no. 6, pp. 111–117, 2026, doi:https://doi.org/10.1109/MCOM.2025.0002230. DOI: https://doi.org/10.1109/MCOM.001.2500230
63. A. Puri, J. Jose, and T. Venkatesh, "Design and Evaluation of a Rack-Scale Disaggregated Memory Architecture for Data Centers," in 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), pp. 212–217, doi:https://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys.2022.00060. DOI: https://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00060
64. A. Puri, K. Bellur, K. Nareddy, J. Jose, V. Tamarapalli, and V. Narayanan, "DRBacklight: Simulating CXL-enabled Large-Scale Disaggregated Memory Systems," in Proceedings of the 35th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation, ser. SIGSIM-PADS '24. New York, NY, USA: Association for Computing Machinery, 2024, pp. 3–14, doi:https://doi.org/10.1145/3638569.3636059. DOI: https://doi.org/10.1145/3615979.3656059
65. S. Karani, J. Wimshe, M. Kuhn, S. Sasse, and D. Froeschle, "NVM in Data Storage: A Post-Optane Future," ACM Trans. Storage, vol. 21, no. 3, Jul. 2025, doi:https://doi.org/10.1145/3731454. DOI: https://doi.org/10.1145/3731454
66. A. Geyer et al., "Near to Far: An Evaluation of Disaggregated Memory for In-Memory Data Processing," in Proceedings of the 1st Workshop on Disruptive Memory Systems, ser. DIMES '23. New York, NY, USA: Association for Computing Machinery, 2023, pp. 16–22, doi:https://doi.org/10.1145/3609308.3625271. DOI: https://doi.org/10.1145/3609308.3625271
67. C. Li, J. Zhao, and Y. Xu, "Efficient Security Support for Cell Memory Through Adaptive Incremental Offloaded (Re-Encryption)," in Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, ser. MICRO '25. New York, NY, USA: Association for Computing Machinery, 2025, pp. 1102–1116, doi:https://doi.org/10.1145/3725843.3756119. DOI: https://doi.org/10.1145/3725843.3756119
68. A. Psistakis, B. Ocalan, C. Alverti, F. Chaix, R. Alagappan, and J. Torrellas, "Towards collision-resilient CXL fabrics," 2026. Online: Available: https://arxiv.org/abs/2602.08261
69. D. Hahlot, V. Khill, L. Werling, T. Gröninger, and F. Bellosa, "Fundamental OS Design Considerations for CXL-based Hybrid SSDs," in Proceedings of the 2nd Workshop on Disruptive Memory Systems, ser. DIMES '24. New York, NY, USA: Association for Computing Machinery, 2024, pp. 51–59, doi:https://doi.org/10.1145/3698783.3699380. DOI: https://doi.org/10.1145/3698783.3699380
70. Z. Zhu et al., "Incomplete Tailored Partial Failures in a CXL Pod," in Proceedings of the 2nd Workshop on Disruptive Memory Systems, ser. DIMES '24. New York, NY, USA, 2024, pp. 41–50, doi:https://doi.org/10.1145/3698783.3699377. DOI: https://doi.org/10.1145/3698783.3699377
71. B. A. Kitchenham and S. Charters, "Guidelines for performing Systematic Literature Reviews in Software Engineering," Keele University, Tech. Rep. EBSE-2007-01, 2007.
72. B. A. Kitchenham, D. Budgen, and P. Brereton, Evidence-Based Software Engineering and Systematic Reviews. New York, NY, USA: Chapman and Hall/CRC, 2015, doi:https://doi.org/10.1201/b19167. DOI: https://doi.org/10.1201/b19467
73. Y. Shakeel et al., "(Automated) Literature Analysis – Threats and Experiences," in Proceedings of the International Workshop on Software Engineering for Science, ser. SE4Science '18. New York, NY, USA: Association for Computing Machinery, 2018, pp. 20–27, doi:https://doi.org/10.1145/3194736.3194748. DOI: https://doi.org/10.1145/3194747.3194748
74. Standard Performance Evaluation Corporation. (2025) SPEC Benchmarks and Tools. Standard Performance Evaluation Corporation. Online: Available: https://www.spec.org/products/
75. Association for Computing Machinery. (2024) ACM Computing Classification System. Association for Computing Machinery. Online: Available: https://dl.acm.org/ccs
76. B. Williams, J. Leddel, X. Wang, D. Donofrio, and Y. Chen, "CircuTen: A Benchmark Suite for Atomic Memory Operations," in Proceedings of the International Symposium on Memory Systems, ser. MEMSYS '20. New York, NY, USA: Association for Computing Machinery, 2021, pp. 144–157, doi:https://doi.org/10.1145/3422575.3422789. DOI: https://doi.org/10.1145/3422575.3422789
77. G. F. Oliveira et al., "DAMOV: A New Methodology and Benchmark Suite for Evaluating Data Movement Bottlenecks," IEEE Access, vol. 9, pp. 13457–134592, 2021, doi:https://doi.org/10.1109/ACCESS.2021.3100993. DOI: https://doi.org/10.1109/ACCESS.2021.3110993
78. A. Qasem, A. M. Aji, and G. Rodgers, "Characterizing data organization effects on heterogeneous memory architectures," in Proceedings of the International Symposium on Code Generation and Optimization, ser. CGO '17. New York, NY, USA: IEEE, 2017, pp. 160–170, doi:https://doi.org/10.1109/CGO.2017.7863737. DOI: https://doi.org/10.1109/CGO.2017.7863737
79. J. Zhang, N. Beckmann, and J. Tse, "GORDON: Benchmarking Optane DC Persistent Memory Modules on FPGAs," in Proceedings of the 29th Annual International Symposium on Field-Programmable Custom Computing Machines, ser. FCCM '21. New York, NY, USA: IEEE, 2021, pp. 97–105, doi:https://doi.org/10.1109/FCCM51124.2021.00021. DOI: https://doi.org/10.1109/FCCM51124.2021.00019
80. D. Shen, X. Liu, and F. X. Lin, "Characterizing Emerging Heterogeneous Memory," in Proceedings of the International Symposium on Memory Management, ser. ISMM '16. New York, NY, USA: Association for Computing Machinery, 2016, pp. 13–24, doi:https://doi.org/10.1145/2926697.2926702. DOI: https://doi.org/10.1145/2926697.2926702
81. A. Ahmed and K. Shahadin, "Hopscotch: A Micro-benchmark Suite for Memory Performance Evaluation," in Proceedings of the International Symposium on Memory Systems, ser. MEMSYS '19. New York, NY, USA: Association for Computing Machinery, 2019, pp. 167–172, doi:https://doi.org/10.1145/3357626.3357754. DOI: https://doi.org/10.1145/3357526.3357574
82. M. Hyldelund, J. J. J. Ang, J. L. Lowe-Power, and V. Akella, "A Case Against Hardware Managed DRAM Caches for NVRAM-Based Systems," in Proceedings of the International Symposium on Performance Analysis of Systems and Software, ser. ISPASS '21. New York, NY, USA: IEEE, 2021, pp. 194–204, doi:https://doi.org/10.1109/ISPASS51385.2021.00036. DOI: https://doi.org/10.1109/ISPASS51385.2021.00036
83. Z. Wang, X. Zhao, T. Michailidis, S. Swanson, and Y. Wu, "Characterizing and Modeling Non-Volatile Memory Systems," in Proceedings of the 53rd Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO '20. New York, NY, USA: IEEE, 2020, pp. 496–508, doi:https://doi.org/10.1109/MICRO56248.2020.00049. DOI: https://doi.org/10.1109/MICRO50266.2020.00049
84. L. McVoy and C. Staelin, "lmbench: Portable Tools for Performance Analysis," in Proceedings of the USENIX Annual Technical Conference, ser. ATC '96. Santa Clara, CA: USENIX Association, 1996, pp. 279–294.
85. A. A. R. Islam, C. York, and D. Dai, "A performance study of optane persistent memory: from storage data structure perspective," CCF Transactions on High Performance Computing, vol. 4, no. 4, pp. 370–393, 2022, doi:https://doi.org/10.1007/s42514-022-00123-x. DOI: https://doi.org/10.1007/s42514-022-00123-x
86. S. Gugnani, A. Kashyap, and X. Lu, "Understanding the Inconsistencies of Read Persistent Memory," Proceedings of the VLDB Endowment, vol. 14, no. 4, pp. 626–639, Dec. 2020, doi:https://doi.org/10.14778/3436905.3436921. DOI: https://doi.org/10.14778/3436905.3436921
87. H. Yoshioka, Y. Hayamizu, K. Goda, and M. Kitsuregawa, "pmem: A Microbenchmark for Understanding Standardization Cost on Persistent Memory," in Proceedings of the International Conference on Big Data and Smart Computing, ser. BigComp '23. New York, NY, USA: IEEE, 2023, pp. 326–333, doi:https://doi.org/10.1109/BigComp2023.00069. DOI: https://doi.org/10.1109/BigComp57234.2023.00069
88. J. Gómez-Luna, I. El Hajj, I. Fernandez, C. Giannoulas, G. F. Oliveira, and O. Mutlu, "Benchmarking Memory-Centric Computing Systems: Analysis of Real Processing-in-Memory Hardware," in Proceedings of the 18th International Green and Sustainable Computing Conference, ser. IGSC '21. New York, NY, USA, 2021, pp. 1–7, doi:https://doi.org/10.1109/IGSC54211.2021.9651614. DOI: https://doi.org/10.1109/IGSC54211.2021.9651614
89. A. Grafpenin, F. Eberhardt, and A. Polze, "SMOG – An Explicitly Composable Memory Benchmark Suite for Heterogeneous Memory," in Proceedings of the International Symposium on Performance Analysis of Systems and Software, ser. ISPASS '23. New York, NY, USA: IEEE, 2023, pp. 113–119, doi:https://doi.org/10.1109/ISPASS50063.2023.00022. DOI: https://doi.org/10.1109/CANDAR60563.2023.00022
90. I. L. Picoli, C. V. Pasco, B. T. Johnson, L. Bounant, and P. Bonnet, "uFLIP-OC: Understanding Flash I/O Patterns on Open-Channel Solid-State Drives," in Proceedings of the 17th Asia-Pacific Workshop on Systems, ser. APSys '17. New York, NY, USA: Association for Computing Machinery, 2017, doi:https://doi.org/10.1145/3124800.3124741. DOI: https://doi.org/10.1145/3124680.3124741
91. S. Che et al., "Rodinia: A Benchmark Suite for Heterogeneous Computing," in Proceedings of the International Symposium on Workload Characterization, ser. IISWC '09. New York, NY, USA: IEEE, 2009, pp. 44–54, doi:https://doi.org/10.1109/IISWC.2009.5306797. DOI: https://doi.org/10.1109/IISWC.2009.5306797
92. R. Shinde, A. Goel, P. Gupta, and D. Dutta, "Similarity search and locality sensitive hashing using ternary content addressable memories," in Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, ser. SIGMOD '10. New York, NY, USA: Association for Computing Machinery, 2010, pp. 375–386, doi:https://doi.org/10.1145/1807167.1807209. DOI: https://doi.org/10.1145/1807167.1807209
93. C. Bienia, S. Kumar, J. P. Singh, and K. Li, "The PARSEC benchmark suite: characterization and architectural implications," in Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques, ser. PACT '08. New York, NY, USA: Association for Computing Machinery, 2008, pp. 72–81, doi:https://doi.org/10.1145/1454115.1454128. DOI: https://doi.org/10.1145/1454115.1454128
94. F. Lalchhandami, K. Datta, S. Chakraborty, R. Dreschler, and I. Sengupta, "CoNMiC: Complementary Memristor based in-memory computing in 3D architecture," Journal of Systems Architecture, vol. 126, p. 102480, 2022, doi:https://doi.org/10.1016/j.sysarc.2022.102480. Online: Available: https://www.sciencedirect.com/science/article/pii/S1383762122000613 DOI: https://doi.org/10.1016/j.sysarc.2022.102480
95. Y. Li et al., "In-Memory Computing using Memristor Arrays with Ultrathin 2D PdSeO2/PdSe2 Heterostructure," Advanced Materials, vol. 34, no. 26, p. 2201488, 2022, doi:https://doi.org/10.1002/adma.202201488. Online: Available: https://advanced.onlinelibrary.wiley.com/doi/abs/10.1002/adma.202201488
96. N. Ihde et al., "A Survey of Big Data, High Performance Computing, and Machine Learning Benchmarks," in Proceedings of the 13th TPC Technology Conference on Performance Evaluation and Benchmarking, ser. TPCTC '21. Cham: Springer International Publishing, 2021, pp. 98–118, doi:https://doi.org/10.1007/978-3-030-94437-7_7. DOI: https://doi.org/10.1007/978-3-030-94437-7_7
97. B. Varghese et al., "A Survey on Edge Performance Benchmarking," ACM Computing Surveys, vol. 54, no. 3, pp. 1–36, Apr. 2021, doi:https://doi.org/10.1145/3444692. DOI: https://doi.org/10.1145/3444692
98. F. Bajaber, S. Sakr, O. Batarfi, A. Altalhi, and A. Barnawi, "Benchmarking big data systems: A survey," Computer Communications, vol. 149, pp. 241–251, 2020, doi:https://doi.org/10.1016/j.comcom.2019.10.002. DOI: https://doi.org/10.1016/j.comcom.2019.10.002
99. R. Han, L. K. John, and J. Zhan, "Benchmarking Big Data Systems: A Review," IEEE Transactions on Services Computing, vol. 11, no. 3, pp. 580–597, 2018, doi:https://doi.org/10.1109/TSC.2017.2730882. DOI: https://doi.org/10.1109/TSC.2017.2730882
100. V. Reniers, D. Van Landuyt, A. Rafique, and W. Joosen, "On the State of NoSQL Benchmarks," in Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion, ser. ICPE '17 Companion. New York, NY, USA: Association for Computing Machinery, 2017, pp. 107–112, doi:https://doi.org/10.1145/3053000.3053622. DOI: https://doi.org/10.1145/3053600.3053622
101. A. Traeger, E. Zadok, N. Joukov, and C. P. Wright, "A Nine Year Study of File System and Storage Benchmarking," ACM Transactions on Storage, vol. 4, no. 2, pp. 1–56, 2008, doi:https://doi.org/10.1145/1362622.1362681. DOI: https://doi.org/10.1145/1367829.1367831
102. L. Wang et al., "BigDataBench: a Big Data Benchmark Suite from Internet Services," in Proceedings of the 20th International Symposium on High Performance Computer Architecture, ser. HPCA '14. New York, NY, USA: IEEE, 2014, pp. 488–499, doi:https://doi.org/10.1109/HPCA.2014.6835958. DOI: https://doi.org/10.1109/HPCA.2014.6835958
Published
Issue
Section
License
Copyright (c) 2026 The Authors. Published by BenchCouncil Press on Behalf of International Open Benchmark Council
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
