Introduction to cross-study research
This section introduces the concept of cross-study research and why it is important.
It also outlines some of the challenges and considerations when carrying out this type of research.
Cross-study research compares the results of a particular research question across studies from different places, from different time periods, or of different ages/generations of people.
Cross-study analysis helps us understand more about group differences, the role of contextual factors, and even how societal changes may impact on outcomes for individuals.
Comparable measures across studies can also potentially allow us to combine data and answer research questions with more precision. Such comparisons offer particular research utility in the context of longitudinal population studies where we want to evaluate and understand change over the lifecourse or over generations.
For example, analysis carried out using the CLOSER harmonised dataset on body composition showed that each generation since 1946 has been heavier than the previous one
What are the challenges of cross-study research?
Cross-study research has many benefits but is often challenging in practice.
A common challenge is that studies tend to use different ways to measure similar things, and it is important to be sure that any differences found between the studies are not artefacts of this different measurement method.
The potential differences between the studies of interest need to be considered at each stage of the project to ensure that findings from the comparisons are valid.
Considerations to make when carrying out cross-study research
(adapted from Bann et al. (2022)).
- What type(s) of studies will be included (e.g. longitudinal, cross-sectional, repeated cross-sectional)?
- Are the target populations comparable?
- Are the sampling designs comparable?
- Do you need to harmonise any variables? Which method is the most appropriate?
- Can you be sure that any change seen is not a methodological artefact? Or could they be due to some of the reasons in the questions below?
- Are the measurement instruments the same (e.g. survey questions, blood pressure monitor)?
- Is there the same amount of measurement error with each instrument?
- Could the responses to the same instrument change over time or between studies (e.g. due to changes in identification or awareness of a condition, or interpretation of the question)?
- Are there differences in the distribution of the study sample across key variables? (e.g. education attainment levels have increased over time in the UK)?
- Do differences between the measurement instruments need to be assessed with measurement invariance tests?
- Is the analysis method used comparable between the studies? Will the data be pooled for analysis, or will the analysis be coordinated across multiple teams?
- Does the sampling design in the studies need to be accounted for in the analyses (e.g. weighting)?
- Are there differences in patterns and magnitudes of missing data in the studies? Does this change the representativeness and comparability?
- Which (if any) missing data methods could be used to maintain statistical power, restore representativeness, and reduce bias?
- Should changes in the results be examined on the relative or absolute scale, or both?
- How will cross-study differences in association be compared? Formally or informally?
- What are the potential sources of differences in the results across the studies? Is there a substantive explanation or could they be due to confounding, sample design, or another reason?
- How do the included studies (type, number, coverage/span) influence the interpretation and generalisability of the results?
- Can the results be interpreted in the same way across the studies (e.g. there could be a causal effect in one study but a confounded effect in another)?
- Are some of the effects context specific? Are they dependent on societal influences that are different in the different studies?
Data harmonisation is way to adjust the data from different studies so that it is comparable. Harmonisation can be done in several different ways, each with their own advantages and disadvantages, but all require significant time and effort to produce comparable data.