CRITERIA, REQUIREMENTS, AND METRICS FOR THE RefMSBench BENCHMARK FOR MICROSERVICE ARCHITECTURE RECONSTRUCTION
DOI:
https://doi.org/10.18372/2310-5461.71.21421Keywords:
microservice architecture, benchmarking, software architecture reconstruction, reference modelAbstract
The paper addresses the problem of selecting a microservice benchmark for evaluating software architecture reconstruction methods. Such evaluation requires more than a deployable and executable system; its structure, interactions, telemetry, and reference model must be comparable with the results produced by reconstruction tools. A set of core and extended criteria is proposed to assess the suitability of MSA benchmarks for reconstruction tasks. The criteria are applied to Sock Shop, TeaStore, DeathStarBench, and Train Ticket. The core criteria cover architectural complexity, telemetry, reproducibility, realism, and evaluability. The extended criteria refine maintainability, configurability, heterogeneity, evolution support, suitability of the reference model for automated verification, and explicit typing of interactions. The analysis shows that the evaluated systems have individual properties useful for architecture reconstruction, but none combines these properties within a single supported evaluation platform. Some benchmarks better represent realistic usage scenarios or provide telemetry, while others offer larger topologies or stronger reproducibility. However, none provides the full set of properties required to evaluate architecture reconstruction results. Based on the analysis, nine functional and fifteen non-functional requirements are formulated for RefMSBench. These requirements cover domain services, an API Gateway, telemetry, the reference model, workload scenarios, and deployment configurations. The structure of the reference model is also defined: it captures services, dependencies, contracts, interaction types, and data ownership. Reconstruction results are evaluated using Service-F1, Edge-F1, and Type-accuracy, together with rules for matching a reconstructed graph against the reference model. The rules specify how services, dependencies, interaction directions, and interaction types should be compared. Future work will focus on the implementation of RefMSBench and the empirical comparison of reconstruction tools.
References
Dragoni N., Giallorenzo S., Lafuente A. L., Mazzara M., Montesi F., Mustafin R., Safina L. Microservices: Yesterday, today, and tomorrow // Present and Ulterior Software Engineering. 2017. С. 195–216. – https://doi.org/10.1007/978-3-319-67425-4_12
Jamshidi P., Pahl C., Mendonça N. C., Lewis J., Tilkov S. Microservices: The Journey So Far and Challenges Ahead. IEEE Software. 2018. Vol. 35, No. 3. P. 24–35. https://doi.org/10.1109/MS.2018.2141039
Wang Y., Kadiyala H., Rubin J. Promises and challenges of microservices: an exploratory study // Empirical Software Engineering. – 2021. – Vol. 26, № 4. – Article 63. – https://doi.org/10.1007/s10664-020-09910-y
Söylemez M., Tekinerdogan B., Kolukısa Tarhan A. Challenges and solution directions of microservice architectures: A systematic literature review // Applied Sciences. – 2022. – Vol. 12, № 11. – Article No. 5507. – https://doi.org/10.3390/app12115507
Bakhtin A., Li X., Soldani J., Brogi A., Cerny T., Taibi D. Tools Reconstructing Microservice Architecture: A Systematic Mapping Study. Software Architecture. ECSA 2023 Tracks, Workshops, and Doctoral Symposium. Cham: Springer, 2024. Lecture Notes in Computer Science. Vol. 14590. P. 3–22. https://doi.org/10.1007/978-3-031-66326-0_1
Ducasse S., Pollet D. Software Architecture Reconstruction: A Process-Oriented Taxonomy. IEEE Transactions on Software Engineering. 2009. Vol. 35, No. 4. P. 573–591. https://doi.org/10.1109/TSE.2009.19
Zhou X., Peng X., Xie T. Benchmarking Microservice Systems for Software Engineering Research. Proceedings of the 40th ACM/IEEE International Conference on Software Engineering (ICSE 2018). New York: Association for Computing Machinery, 2018. P. 323–324. https://doi.org/10.1145/3183440.3194991
Smith S., Robinson E., Frederiksen T., Stevens T., Cerny T., Bures M., Taibi D. Benchmarks for End-to-End Microservices Testing. Proceedings of the 17th IEEE International Conference on Service-Oriented System Engineering (SOSE 2023). Piscataway: IEEE, 2023. P. 60–66. https://doi.org/10.1109/SOSE58276.2023.00013
Gan Y. et al. An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems. Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2019). New York: ACM, 2019. P. 3–18. https://doi.org/10.1145/3297858.3304013
Granchelli G. et al. MicroART: A Software Architecture Recovery Tool for Maintaining Microservice-Based Systems. Proceedings of the 2017 IEEE International Conference on Software Architecture Workshops (ICSAW 2017). Gothenburg: IEEE, 2017. P. 298–302. https://doi.org/10.1109/ICSAW.2017.9
Al Maruf A., Bakhtin A., Cerny T., Taibi D. Using Microservice Telemetry Data for System Dynamic Analysis. Proceedings of the 2022 IEEE International Conference on Service-Oriented System Engineering (SOSE 2022). Piscataway: IEEE, 2022. P. 29–38. https://doi.org/10.1109/SOSE55356.2022.00010
Waseem M., Liang P., Shahin M., Di Salle A., Márquez G. Design, Monitoring, and Testing of Microservices Systems: The Practitioners’ Perspective. Journal of Systems and Software. 2021. Vol. 182. Article 111061. https://doi.org/10.1016/j.jss.2021.111061
Hutcheson R., Blanchard A., Lambaria N., Hale J., Kozak D., Abdelfattah A. S., Cerny T. Software Architecture Reconstruction for Microservice Systems Using Static Analysis via GraalVM Native Image. Proceedings of the 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2024). Piscataway: IEEE, 2024. P. 12–22. https://doi.org/10.1109/SANER60148.2024.00008
Bushong V., Das D., Cerny T. Reconstructing the Holistic Architecture of Microservice Systems Using Static Analysis. Proceedings of the 12th International Conference on Cloud Computing and Services Science (CLOSER 2022). Science and Technology Publications, 2022. P. 149–157. https://doi.org/10.5220/0011032100003200
Schneider S., Bakhtin A., Li X., Soldani J., Brogi A., Cerny T., Scandariato R., Taibi D. Comparison of Static Analysis Architecture Recovery Tools for Microservice Applications. Empirical Software Engineering. 2025. Vol. 30, No. 5. Article 128. https://doi.org/10.1007/s10664-025-10686-2
Sedghpour M. R. S. et al. HydraGen: A Microservice Benchmark Generator. Proceedings of the 2023 IEEE 16th International Conference on Cloud Computing (CLOUD 2023). Piscataway: IEEE, 2023. P. 189–200. https://doi.org/10.1109/CLOUD60044.2023.00030
Bliudze S. et al. Adaptable TeaStore. Proceedings of the First Workshop on Adaptable Cloud Architectures (WACA 2025). Open Publishing Association, 2025. P. 1–14. https://doi.org/10.4204/EPTCS.438.1
Моргун А., Гізун А. Концепт програмного забезпечення для проєктування мікросервісної архітектури з використанням штучного інтелекту. Вісник Хмельницького національного університету. Технічні науки. 2025. № 5 (357). С. 307–313. https://doi.org/10.31891/2307-5732-2025-357-39
Von Kistowski J. et al. TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research. Proceedings of the 26th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2018). IEEE, 2018. P. 223–236. https://doi.org/10.1109/MASCOTS.2018.00030
Garlan D., Monroe R., Wile D. ACME: An Architecture Description Interchange Language. Proceedings of the 1997 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON 1997). Toronto: IBM Press, 1997. P. 169–183.
Lutellier T. et al. Comparing Software Architecture Recovery Techniques Using Accurate Dependencies. Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE 2015). Piscataway: IEEE, 2015. P. 69–78. https://doi.org/10.1109/ICSE.2015.136
Garcia J., Ivkovic I., Medvidovic N. A Comparative Analysis of Software Architecture Recovery Techniques. Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering (ASE 2013). Piscataway: IEEE, 2013. P. 486–496. https://doi.org/10.1109/ASE.2013.6693106
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Artem Morhun, Андрій Гізун
This work is licensed under a Creative Commons Attribution 4.0 International License.
The scientific journal adheres to the principles of Open Access and provides free, immediate, and permanent access to all published materials without financial, technical, or legal barriers for readers.
All articles are published in Open Access under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.
Copyright
Authors who publish their works in the journal:
-
retain the copyright to their publications;
-
grant the journal the right of first publication of the article;
-
agree to the distribution of their materials under the CC BY 4.0 license;
-
have the right to reuse, archive, and distribute their works (including in institutional and subject repositories), provided that proper reference is made to the original publication in the journal.
