Addressing sample overlap using Generalized Weights should now be straightforward

Heiko Rachinger and I recently got our paper on addressing sample overlap in economics meta-analyses accepted for publication in the Journal of Economic Surveys (see here). The paper demonstrates how to apply our generalized-weights (GW) method (developed in a paper published in Research Synthesis Methods in 2020, see here) in cases of data aggregation, different estimation methods, or conversion to PCCs. By tackling these challenges, we hope to make the application of GW relatively straightforward.

For those unfamiliar with our GW method, it incorporates the covariance between sample-overlapping estimates in their weights in the meta-analysis. An estimate is given less weight not only when its variance is large, as in conventional inverse-variance weights, but also when its covariance with another estimate is large.

Thus far, meta-analysts have addressed sample overlap mostly by using clustered standard errors or multilevel models. These methods are satisfactory in same cases—e.g., when sample confined at the study level—but cannot completely address sample overlap in other cases. GW can tackle any case of sample overlap, within or between studies, no matter how complex it may be. Moreover, GW can easily be combined with those methods, if necessary to tackle estimate dependence caused by other factors (e.g., study-specific effects).

To make the implementation of GW as easy as possible, we have prepared Stata and R codes, which are publicly available on OSF (see here). They include companion documents with instructions for implementation. It's especially important to note how the input file must be structured for the codes to run smoothly.

Our method is very general and can tackle virtually any type of sample overlap. In cases where sample overlap takes different forms than those addressed in the paper, it may be necessary to tweak the input file slightly. With some ingenuity, we believe that virtually any sample overlap case can be addressed.

We invite you all to give it a try! If you encounter problems, please contact us at pedro.bom@deusto.es or heiko.rachinger@uib.es. Thanks!