Introduction to cross-study research

Cross-study research compares the results of a particular research question across studies from different places, from different time periods, or of different ages/generations of people.

Cross-study analysis helps us understand more about group differences, the role of contextual factors, and even how societal changes may impact on outcomes for individuals.

Comparable measures across studies can also potentially allow us to combine data and answer research questions with more precision. Such comparisons offer particular research utility in the context of longitudinal population studies where we want to evaluate and understand change over the lifecourse or over generations.

For example, analysis carried out using the CLOSER harmonised dataset on body composition showed that each generation since 1946 has been heavier than the previous one.

• What type(s) of studies will be included (e.g. longitudinal, cross-sectional, repeated cross-sectional)?
• Are the target populations comparable?
• Are the sampling designs comparable?

• Do you need to harmonise any variables? Which method is the most appropriate?
• Can you be sure that any change seen is not a methodological artefact? Or could they be due to some of the reasons in the questions below?
• Are the measurement instruments the same (e.g. survey questions, blood pressure monitor)?
• Is there the same amount of measurement error with each instrument?
• Could the responses to the same instrument change over time or between studies (e.g. due to changes in identification or awareness of a condition, or interpretation of the question)?
• Are there differences in the distribution of the study sample across key variables? (e.g. education attainment levels have increased over time in the UK)?
• Do differences between the measurement instruments need to be assessed with measurement invariance tests?

• Is the analysis method used comparable between the studies? Will the data be pooled for analysis, or will the analysis be coordinated across multiple teams?
• Does the sampling design in the studies need to be accounted for in the analyses (e.g. weighting)?
• Are there differences in patterns and magnitudes of missing data in the studies? Does this change the representativeness and comparability?
• Which (if any) missing data methods could be used to maintain statistical power, restore representativeness, and reduce bias?
• Should changes in the results be examined on the relative or absolute scale, or both?
• How will cross-study differences in association be compared? Formally or informally?

• What are the potential sources of differences in the results across the studies? Is there a substantive explanation or could they be due to confounding, sample design, or another reason?
• How do the included studies (type, number, coverage/span) influence the interpretation and generalisability of the results?
• Can the results be interpreted in the same way across the studies (e.g. there could be a causal effect in one study but a confounded effect in another)?
• Are some of the effects context specific? Are they dependent on societal influences that are different in the different studies?