This is the code repository for the reproducibility study conducted on the following case study. The original codebase was downloaded and copied to a new repository to re-execute and computationally reproduce the results.
Case Study: Kieu, L.-M., Malleson, N., & Heppenstall, A. (2020). Dealing with uncertainty in agent-based models for short-term predictions. Royal Society Open Science, 7(1), 191074. https://doi.org/10.1098/rsos.191074 URL: https://royalsocietypublishing.org/author/Heppenstall%2C+Alison
Case Study Codebase: leminhkieu/Bus-Simulation-model: Full version of BusSim, including synthetic data URL: https://zenodo.org/records/3549633 File downloaded: leminhkieu/Bus-Simulation-model-v1.0.1.zip Date downloaded: 07/04/2025
The results of this investigation have been documented in the following works:
Conference Paper: Add conference paper information
Master's Thesis: Tung, Gabriella Low Chew 2025. “Computational Reproducibility in Modelling and Simulation Studies: A Case Study on Data Assimilation Algorithms for Agent-based Modelling and Simulation”. Delft University of Technology, the Netherlands. URL: https://resolver.tudelft.nl/uuid:e20468fa-7055-4d6c-9627-7a4f98b9e41e
This study's codebase is also archived on the Zenodo platform:
Reproducibility Study's codebase Tung, Gabriella Low Chew 2026, April. “5690-cosh/test-bus-sim: Alpha Version” https://doi.org/10.5281/zenodo.19367488.
The folder structure of this repository is as follows:
├───Documentation
├───leminhkieu-Bus-Simulation-model-fd82a47
│ ├───Calibration
│ ├───Data
│ ├───experiment1a_fig5
│ ├───experiment1a_fig6
│ ├───Figures
│ ├───original
│ │ ├───originalcalibration
│ │ ├───originaldata
│ │ └───originalfigures
├───leminhkieu-Bus-Simulation-model-fd82a47_newdownload
│ ├───Calibration
│ ├───Data
│ ├───Figures
│ ├───figure_7
│ ├───figure_8
│ │ └───bs_stochastic
│ ├───figure_9
│ ├───sensitivity_analysis
│ │ ├───Data
│ │ └───pf_exp_results_rep
│ │ └───rmse_figures
│ ├───table3_v2
│ │ ├───Calibration
│ │ ├───Data
│ │ ├───final_results
│ │ ├───scenario1
│ │ ├───scenario2
│ │ └───scenario3
└───paper_figures
└───wpd_data_for_RMSE
├───figure 7
├───figure 8
└───figure 9
This repository contains two copies of the case study's original codebase.
This folder is used to reproduce Figures 5 and 6 from the case study's publication.
The python notebook 'leminhkieu-Bus-Simulation-model-fd82a47\experiment1a_fig5.ipynb'is used to reproduce Figure 5 with output files stored in the folder 'leminhkieu-Bus-Simulation-model-fd82a47\experiment1a_fig5'.
The python notebook 'leminhkieu-Bus-Simulation-model-fd82a47\experiment1a_fig6.ipynb' is used to reproduce Figure 6 with output files stores in the folder 'leminhkieu-Bus-Simulation-model-fd82a47\experiment1a_fig6'.
This folder contains the scripts used to reproduce Figures 7,8, 9 and Table 3 from the case study publication. Further experimentation was also conducted on the particle filter implementation ('Scenario 3'). The subdirectories are described below.
'leminhkieu-Bus-Simulation-model-fd82a47_newdownload\figure_7' This folder contains the scripts needed needed to reproduce Figure 7 of the case study. This corresponds to Figure 8 of the Master's Thesis.
leminhkieu-Bus-Simulation-model-fd82a47_newdownload\figure_8 This folder contains the scripts needed needed to reproduce Figure 8 of the case study. This corresponds to Figure 10 of the Master's Thesis.
leminhkieu-Bus-Simulation-model-fd82a47_newdownload\figure_9 This folder contains the scripts needed needed to reproduce Figure 9 of the case study. This corresponds to Figure 12 of the Master's Thesis.
leminhkieu-Bus-Simulation-model-fd82a47_newdownload\table3_v2 This folder contains the scripts needed needed to reproduce Table 3 - Sensitivity Analysis. This corresponds to Tables 5 and 6 and Figures 13 and 14 in the Master's Thesis. This folder reproduces the results of Table 1(b).
leminhkieu-Bus-Simulation-model-fd82a47_newdownload\sensitivity_analysis
This folder contains the scripts needed needed to reproduce the extended experimentation of the particle filter.
This corresponds to Figures 17 to 24 in the Master's Thesis.
This folder is used to reproduce Figures 1 and 2 of the conference paper.
This folder contains the extracted data points from the case study publication figures. The data points were extracted using online software WebPlotDigitizer. This folder reproduces the results of Table 1(a). It includes the tar files produced via the data extraction process. These files can be opened by uploading them to WebPlotDigitzer platform.
Rohatgi, A. WebPlotDigitizer. In (Version 5.2) https://automeris.io
This folder contains logs noting the challenges/runtime errors encountered throughout the reproduction process. It also includes a workflow diagram relating the case study's figures to the codebase scripts required for reproduction.
See 'change-error log.docx' and 'Error_log.md' to see notes on runtime errors encountered when rerunnning the original repository.
See the word document 'postGL_experiments.docx' and 'Experiment Log.docx' to see documentation on the process and assumptions made for experiments.
Python version: Python 3.13.1
To rerun this reproducibility experiment, first setup the virtual environment using the 'requirements.txt' file.
Hardware used:
| Hardware used | Description |
|---|---|
| System | LENOVO ThinkPad P14s Gen 5 AMD |
| Processor | AMD Ryzen 7 PRO 8840HS w/ Radeon 780M Graphics, 3301 Mhz, 8 Core(s), 16 Logical Processor(s) |
| RAM | 16 GB |
| Software used | Description |
|---|---|
| Operating System | Microsoft Windows 11 Home |
| Operating System Version | 10.0.26100 Build 26100 |
| Integrated Development Environment (IDE) | Visual Studio Code May 2025 (version 1.101.2) Release date: June 12, 2025 |
| Python | Python 3.13.1 |
| Package Manager | pip 25.1.1 |
| Provider of pseudo-random number generator | Numpy 2.2.4 |
Further details can be found in the README.md in the experiment folders or in the specific python notebooks.