Computational models predicting the early development of the COVID-19 pandemic in Sweden: systematic review, data synthesis, and secondary validation…

The study was conducted as a systematic review of published literature followed by a data synthesis6,7. For this purpose, searches were carried out for scientific publications (scientifically reviewed before publication), preprints (i.e. articles of a scientific nature that are published openly without prior review) and the gray literature (i.e. reports and documents published by organizations and authorities). The study protocol is registered in the database for structured literature syntheses and meta-analyzes PROSPERO (International prospective register of systematic reviews) no. CRD42021229514 (see Supplement S1).

The literature searches were based on the search triangle model6. Systematic searches were conducted between 22 January 2021 and 29 January 2021 of databases (PubMed, Cochrane Library, Embase, Love platform/Epistemikos), containing peer-reviewed scientific publications and systematic reviews in areas relevant to the review issue, exploratory searches were performed in preprint archives, while look-up searches were performed in the gray literature. The literature searches were reported according to the PRISMA-S protocol (see Supplement S2).

The systematic search (keywords: prediction, nowcast, forecast, simulation model, model, modeling, estimation, scenario, surveillance, Epidemiology, COVID-19, SARS-cov-2, swed*) of the collegially assessed scientific literature had the goal to identify all relevant publications (within the criteria of the study) in a transparent and reproducible manner.

The explorative searches in the preprint archives were initiated by asking a preliminary question via a tool specifically designed for searches in these archives (search.biopreprint) and then reviewing the recovered records. Thereafter, the searches were repeated iteratively until adjustments no longer led to significant changes in the set of identified preprints. A separate supplementary search was performed against the two largest preprint databases bioRxiv (which also includes preprints from medRxiv) and arXiv. Finally, a search (directed search) of the gray literature was performed. The searchalso called search for known documentswas carried out with the aim of obtaining documents from the websites of relevant Swedish and international authorities active in the area: PHAS, the National Board of Health and Welfare, the Swedish Civil Contingencies Agency and the European Center for Disease Prevention and Control (ECDC). Local and regionally produced forecast data in different healthcare regions are not included in this report. These are regarded as internal working material since they are not published and not publicly available.

scientific articles that report epidemiological results regarding actual or scenario-based predictions of morbidity, mortality, or healthcare burden caused by COVID-19 in Sweden or parts of Sweden in 2020.

reports of COVID-19 modelling published by the PHAS.

non-original analyzes (e.g. reviews, perspective articles, editorials, recommendations and guidelines).

duplicate studies.

in silico studies (pure simulations without comparison with data).

descriptive epidemiological publications (e.g. description of case incidences and geographical distributions).

models that only examine the effect of interventions (rather than predicting risk or disease burden).

articles or reports that present new mathematical models or software tools, unless an explicit central purpose of the study is to predict COVID-19 phenomena.

articles or reports from which predictions could not be extracted as a time series.

articles or reports that present predictions that are adjacent to or fall completely outside of 2020.

The systematic searches in the peer-reviewed scientific literature, the exploratory searches of preprint archives and the look-up searches in the gray literature resulted in document material being examined prior to data extraction. In this inclusion-confirming step, titles and summaries of the documents obtained were reviewed against the study criteria (inclusion/exclusion) by two independent reviewers. Documents that both reviewers considered to be included were included and those that both excluded were excluded from further analysis. In case of disagreement, the articles were downloaded in full text and a new assessment was made. If the disagreement persisted, this was resolved through discussions between the reviewers and, if necessary, with the research group. For data extraction from the final set of documents, a tool for retrieving data from each article in full text was developed. The tool included data on the authors' country of origin, study design, forecast methodology (type of model), study population, data sources, forecast period, forecast results, measures of prediction accuracy/performance (if applicable) and model documentation. One reviewer initially extracted data from each included article and then two other reviewers checked the data obtained. The data extracted from the articles were documented in a spreadsheet.

All models were assessed for systematic sources of error (bias). In articles that addressed several models, each model was assessed separately. For the assessment, a form, ROBOT (Risk of Bias Opinion Tool), was developed, based on previous guidelines for evaluations of forecast studies8,22. In summary, the following topics were examined at model level: relevance and quality of data, time frame for prediction, assumptions, and model development methods (verification and validation). The assessment of assumptions included reproduction rates, latency period, incubation period, serial interval, infectious period, population immunity, and impact of interventions during the prediction period. Model validation was classified as one out of three: retrospective/internal validation, external validation, or no validation.

The assessment of systematic sources of error was performed by two independent assessors, where another assessor assisted in case of disagreement. Each sub-aspect was given a score rating in an assessment form, ROBOT, (see Supplement S3). The partial assessments were added up to a total score for each model. To qualify for further result synthesis, a total score below a heuristically defined limit value was required (ROBOT<4). Given the impact of predictions made by PHAS these were included in the result synthesis even if they failed the ROBOT cut-off.

A secondary validation of model performance was made, where reported predictions were compared with factual outcome data. The data on the forecasting variables were retrieved from published figures using WebPlotDigitizer (v. 4.4, https://apps.automeris.io/wpd/). The models in the final set addressed the total incidence of COVID-19 cases, ICU-occupancy, and incidence of COVID-19 deaths. A simultaneous evaluation of prediction accuracy that included all models was not feasible due to differences in study populations, modeled outcome, and time period. The secondary validation was therefore broken down into subgroups based on the reported outcome variables. Data on the actual outcomes on deaths and ICU-occupancy were obtained from PHAS. Regarding the total case incidence, no source for reliable outcome data was available due to the variable testing strategy employed in Sweden during 2020. When possible, the model performance was quantified by measuring the Mean Absolute Percentage Error (MAPE) between model predictions and the outcome for the entire time period covered by each separate model. We classified the performance according to the following scheme: 0%MAPE10%excellent, 10%30%poor. Based on experiences from public health practitioners during the pandemic, as well as the fact that Sweden already before the pandemic lacked healthcare resources (for instance, at average 103 patients share 100 available hospital beds9), these limits was considered reasonable. The dates when the predictions were made (models finally calibrated) were retrieved from the articles. We acknowledge that measures have been developed that avoid some of the drawbacks of MAPE (e.g. divergence for outcomes close to zero)23, but for clarity and interpretability we opted for MAPE. To determine if difference in prediction errors had statistical significance, we employed the Diedbold-Mariano test. This test requires that the predictions are made for the exact same time period, and we therefore applied the test to the intersection of all prediction dates.

Not all predictions of the total number of cases did include entire Sweden, but all included the Stockholm region. The evaluation was therefore restricted to forecasting the pandemic development in this region (population 2.3 million). In order to be able to compare predictions of the total incidence of COVID-19 cases from PHAS, we had to adjust the predictions from PHAS, which are in terms of the number of reported cases. In the reports from PHAS (e.g. 35 in Table 1), the proportion of unconfirmed cases was estimated to be 98.7%, which made it possible to rescale the predictions of reported cases by dividing those predictions by (10.987), and thus obtaining the total number of cases.

All predictions of ICU-occupancy did not include the entire country but did include the Stockholm region. Also, this evaluation was therefore restricted to the Stockholm region. While acknowledging that assumptions regarding epidemiological homogeneity introduce uncertainty, we multiplied the predictions by the proportion of the total Swedish population that lived in the Stockholm region to allow comparisons with the entire country.

We compared predictions of the number of deaths in COVID-19 during the spring of 2020. In relation to this, we also analysed how much historical data was used to calibrate the models in relation to the length of the prediction by calculating the ratio of the number of days of data used in the calibration and the length of the prediction (in days).

See the rest here:

Computational models predicting the early development of the COVID-19 pandemic in Sweden: systematic review, data synthesis, and secondary validation...