Speakers
Description
In the wake of the 2010s replicability debates, current research witnesses unprecedented amounts of replication studies and theorizing about these. Numerous replication metrics exist, but no consensus about what constitutes replication success/failure. As one-to-many (repeated) replication designs increasingly are the norm, we utilize a combinatorial (all-subsets) meta-analysis (CMA) framework, representing a suitable thought model for the entire research process: other subsets of replications could have been conducted than those known; other study subsets could be included in meta-analyses. Consequently, we propose (and illustrate with a real-world research claim) a set of novel, CMA-based replication metrics and displays: (1) majority vote from a CMA of replication outcomes; (2) percentile location of the original study’s effect size (ES) within the summary ES distribution from such a CMA; (3) median ES (plus interquartile range) and (4) average ES (with 5th and 95th percentiles) of this distribution; (5) a traffic-light bar plot, visualizing the proportions of significant/nonsignificant CMA summary ES which are directionally congruent/opposite to the original study’s outcome.
