RSRBENCH: REPRODUCIBILITY AND PERFORMANCE BENCHMARKING FOR REGIONAL SEISMIC RISK WORKFLOWS
Keywords:
Regional seismic risk, Reproducibility, Benchmarking, Scientific workflows, Parallel random streams, Fault tolerance, NHERI SimCenterAbstract
Regional seismic risk assessment involves computational workflows that integrate hazard mapping, structural response analysis, and damage-and-loss estimation. Although these workflows are increasingly executed in parallel, their reproducibility across different execution configurations and failure conditions remains insufficiently characterized. This paper presents RSRBench, a reproducible benchmarking framework for evaluating the consistency, performance, and reliability of regional seismic assessment workflows. Using publicly available San Francisco and Anchorage datasets and the NHERI SimCenter software, we examine the effects of execution parallelism, random-number generation, task scheduling, and failure recovery. An execution-level audit reveals that changing the number of parallel processes alters hazard assignments for 596–598 of 600 evaluated buildings, while applying a single global random seed increases portfolio-loss variability by 38% in the tested configuration. To address these issues, RSRBench introduces asset-specific random-number streams, order-independent aggregation, and task-level recovery mechanisms. Experiments demonstrate identical per-asset results across the tested scheduling and failure-recovery configurations. We further evaluate computational scalability using measured task execution times and a trace-driven simulator validated against process-pool emulation, with a mean absolute prediction error of 3.5%. At 1,024 simulated workers, cost-aware scheduling improves parallel efficiency from 0.88 to 0.99 compared with static cyclic scheduling. These findings establish a reproducible evaluation methodology for improving the consistency and computational efficiency of regional seismic risk workflows.References
[1] Federal Emergency Management Agency. Hazus Earthquake Model Technical Manual, Hazus 6.1. Washington, DC: FEMA, 2024.
[2] Kircher C A, Whitman R V, Holmes W T. HAZUS Earthquake Loss Estimation Methods. Natural Hazards Review, 2006, 7(2): 45-59. DOI: 10.1061/(ASCE)1527-6988(2006)7:2(45).
[3] Federal Emergency Management Agency. Seismic Performance Assessment of Buildings, Volume 1 – Methodology, Second Edition (FEMA P-58-1). Washington, DC: FEMA, 2018.
[4] Lu X, McKenna F, Cheng Q, et al. An Open-Source Framework for Regional Earthquake Loss Estimation Using the City-Scale Nonlinear Time History Analysis. Earthquake Spectra, 2020, 36(2): 806-831.
[5] Deierlein G G, Zsarnóczay A, Singh K, et al. A Cloud-Enabled Application Framework for Simulating Regional-Scale Impacts of Natural Hazards on the Built Environment. Frontiers in Built Environment, 2020, 6: 558706. DOI: 10.3389/fbuil.2020.558706.
[6] Rathje E M, Dawson C, Padgett J E, et al. Enhancing Research in Natural Hazards Engineering through the DesignSafe Cyberinfrastructure. Frontiers in Built Environment, 2020, 6: 547706. DOI: 10.3389/fbuil.2020.547706.
[7] Zsarnóczay A, NHERI SimCenter. pelicun: Probabilistic Estimation of Losses, Injuries, and Community Resilience under Natural Hazard Events, Version 3.8.1. Software release, 2025. NHERI SimCenter. Regional Resilience Determination Tool (R2D), Version 5.5.0 Release Notes. 2025.
[8] Silva V, Crowley H, Pagani M, et al. Development of the OpenQuake Engine, the Global Earthquake Model’s Open-Source Software for Seismic Risk Assessment. Natural Hazards, 2014, 72(3): 1409-1427. DOI: 10.1007/s11069-013-0618-x.
[9] Xiong C, Lu X, Lin X, et al. Parameter Determination and Damage Assessment for THA-Based Regional Seismic Damage Prediction of Multi-Story Buildings. Journal of Earthquake Engineering, 2017.
[10] Jayaram N, Baker J W. Correlation Model for Spatially Distributed Ground-Motion Intensities. Earthquake Engineering & Structural Dynamics, 2009, 38(15): 1687-1708.
[11] Weatherill G, Silva V, Crowley H, et al. Exploring the Impact of Spatial Correlations and Uncertainties for Portfolio Analysis in Probabilistic Seismic Loss Estimation. Bulletin of Earthquake Engineering, 2015, 13(4): 957-981. DOI: 10.1007/s10518-015-9730-5.
[12] Peng R D. Reproducible Research in Computational Science. Science, 2011, 334(6060): 1226-1227. DOI: 10.1126/science.1213847.
[13] Stodden V, McNutt M, Bailey D H, et al. Enhancing Reproducibility for Computational Methods. Science, 2016, 354(6317): 1240-1241. DOI: 10.1126/science.aah6168.
[14] Hoefler T, Belli R. Scientific Benchmarking of Parallel Computing Systems: Twelve Ways to Tell the Masses When Reporting Performance Results. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC15), 2015: 73:1-73:12. DOI: 10.1145/2807591.2807644.
[15] Salmon J K, Moraes M A, Dror R O, et al. Parallel Random Numbers: As Easy as 1, 2, 3. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC11), 2011: 16:1-16:12.
[16] L’Ecuyer P. Multiple Streams with Recurrence-Based, Counter-Based, and Splittable Random Number Generators. Proceedings of the 2021 Winter Simulation Conference (WSC), 2021.
[17] Harris C R, Millman K J, van der Walt S J, et al. Array Programming with NumPy. Nature, 2020, 585(7825): 357-362. DOI: 10.1038/s41586-020-2649-2.
[18] Shewchuk J R. Adaptive Precision Floating-Point Arithmetic and Fast Robust Geometric Predicates. Discrete & Computational Geometry, 1997, 18(3): 305-363. DOI: 10.1007/PL00009321.
[19] Demmel J, Nguyen H D. Fast Reproducible Floating-Point Summation. Proceedings of the 21st IEEE Symposium on Computer Arithmetic (ARITH), 2013: 163-172. DOI: 10.1109/ARITH.2013.9.
[20] Graham R L. Bounds on Multiprocessing Timing Anomalies. SIAM Journal on Applied Mathematics, 1969, 17(2): 416-429. DOI: 10.1137/0117039.
[21] Polychronopoulos C D, Kuck D J. Guided Self-Scheduling: A Practical Scheduling Scheme for Parallel Supercomputers. IEEE Transactions on Computers, 1987, C-36(12): 1425-1439. DOI: 10.1109/TC.1987.5009495.
[22] Young J W. A First Order Approximation to the Optimum Checkpoint Interval. Communications of the ACM, 1974, 17(9): 530-531.
[23] Daly J T. A Higher Order Estimate of the Optimum Checkpoint Interval for Restart Dumps. Future Generation Computer Systems, 2006, 22(3): 303-312. DOI: 10.1016/j.future.2004.11.016.
[24] NHERI SimCenter. R2D User Manual: Regional Earthquake Assessment Examples (E1 Basic HAZUS and E2 MDOF Building Response). R2D v5.5 Documentation, 2025.
[25] McKenna F. OpenSees: A Framework for Earthquake Engineering Simulation. Computing in Science & Engineering, 2011, 13(4): 58-66. DOI: 10.1109/MCSE.2011.66.
[26] Benjamini Y, Hochberg Y. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B, 1995, 57(1): 289-300. DOI: 10.1111/j.2517-6161.1995.tb02031.x.
[27] Zhu M, McKenna F, Scott M H. OpenSeesPy: Python Library for the OpenSees Finite Element Framework. SoftwareX, 2018, 7: 6-11. DOI: 10.1016/j.softx.2017.10.009.