Speakers
Description
Decisions in data analysis often are arbitrary. Addressing such “researcher degrees of freedom”, multiverse analysis is an approach that identifies all reasonable data-analytic specifications and calculates results for all possible combinations thereof. Here, we propose cross-temporal multiverse analysis as an extension, by accounting for the temporal dimension in case of dynamically changing data. We demonstrate its usefulness by reanalyzing disputed research evidence on COVID-19 mortality and testosterone markers. Combining all reasonable specifications, we calculated multiverse analyses for each day from March 2020 to May 2021. On the last day of the observation period, only 6.25% of 192 relationships tested were statistically significant (i.e., largely consistent with mere chance, 5%). Cross-temporal analysis revealed that none of these significant tests were stable over time. Hence, reported evidence for associations of COVID-19 mortality and testosterone markers is insufficient. Cross-temporal multiverse analysis shows promise as an effective method for testing robustness of evidence based on dynamically changing datasets (e.g., during the ongoing COVID-19 pandemic).
