CLOSER Director, Professor Alison Park, reflects on the 2017 Longitudinal Studies Review and argues that work led by CLOSER to enhance these world-class studies by improving their comparability will be vital to their future sustainability.
Last month over a hundred academics met in Milan for the annual conference of the Society of Lifecourse and Longitudinal Studies (SLLS). CLOSER ran a session that focused on the opportunities and obstacles involved in cross-study research, and the data harmonisation that this typically involves.
Data harmonisation is the process by which data from different studies and periods of data collection are made more amenable to comparison. Harmonisation was identified as one of the “top ten” methodological and technological issues in a 2016 online consultation exercise carried out prior to the recent Longitudinal Studies Strategic Review. The Review, commissioned by the ESRC, assessed the continuing scientific needs for longitudinal research resources and offered recommendations on ways of enhancing this portfolio in the future. It too highlighted harmonisation as a key element within any future longitudinal strategy.
The value of harmonisation
Harmonisation can be motivated by many research aims, including a desire to understand the experiences of different generations or to compare patterns of association across studies that cover different populations or life stages. Such uses illustrate how the findings established through individual studies can be further enhanced when we can compare results from across multiple studies and contexts.
Cross-study work can yield valuable insights, providing the richness and depth of information to answer scientific and policy questions that simply was not possible in the past. For example, CLOSER funded work to harmonise measures of BMI has shown that each generation since 1946 has been heavier than the previous one – and it is the most overweight people who are becoming even heavier. The work, cited in the Government’s Childhood Obesity Plan, also showed that people are becoming overweight or obese at an increasingly younger age.
CLOSER has funded an array of work in this area. Previous work has included harmonising measures of socio-economic status and qualifications, senses and behaviours and overcrowding. We are also funding new research projects that seek to harmonise measures including physical activity, dietary data, cognitive measures and mental health measures. The latter will investigate and compare the development of common mental disorders over the life course in different generations, as well as test whether mental health is improving or declining in more recently born cohorts.
Harmonisation challenges
The actual data harmonisation process is rarely simple. At CLOSER’s SLLS symposium we identified four key challenges.
- Harmonisation is necessary because different studies use different questions, techniques or devices to measure the same concepts. At times this may reflect changing practice, theory or technology – but there can also be disciplinary differences in approach. Sometimes the variation is so considerable that harmonisation is simply not possible; others require considerable effort to overcome. Ultimately, the ambition is to be sure that any differences you find are not simply artefacts of the way different studies have collected data. However, as there is no simple way of validating your approach, being confident you have identified the optimum approach requires time and careful consideration.
- Even when the precise measure remains the same (for example, whether or not someone grew up in a home with an outside toilet), its meaning now may be very different to its meaning in the past. This is a particular challenge for those who want to compare the experiences of different generations to see how society is changing over time.
- Any harmonisation of longitudinal data requires an understanding of the different factors that underpin missing data, and being able to make informed decisions about the most appropriate strategy to take account of it.
- Finally, harmonisation requires a huge amount of effort – and the rewards, which often only emerge once a key paper is published, do not always seem commensurate.
How CLOSER is helping
At CLOSER we have produced a range of useful scientific outputs for researchers interested in cross-study research. These both demonstrate the value of harmonisation and help others who wish to make use of the harmonised outputs as well as undertake harmonisation of their own.
As the previous discussion about its challenges suggests, researchers who are interested in harmonisation require as much information as possible about the data they are intending to use. Discovery is a resource developed by CLOSER to help researchers find out detailed information about the data that is out there. We are now adding functionality to allow the identification of ‘equivalent’ questions and variables across studies to help with harmonisation and cross-study research.
Given the time consuming nature of harmonisation, wherever possible others should be able to benefit from work that has already been done. CLOSER’s many data harmonisation projects typically produce harmonised and fully documented datasets (which are made freely available to researchers via the UK Data Service) as well as journal articles and other research outputs.
CLOSER also runs workshops and training courses on a range of subjects of relevance to longitudinal and inter-study research efforts. We will be running a workshop on the practice and management of cross-study comparisons at the University of Manchester on the 16th October this year.
CLOSER’s aim throughout these activities is to help to augment the value of existing datasets and facilitate more cross-study research for generations to come. Our work will also, we hope, reduce the need for extensive retrospective harmonisation by helping inform the choices studies make about how they measure key concepts in the future.
“The ESRC needs to take advantage of opportunities (such as harmonisation) to help ensure that the UK maintains its leading position in this area in the future.” 2017 Longitudinal Studies Review
Suggested citation:
Park A (2018) “Cross-study comparisons: The value and challenges of data harmonisation”, CLOSER blog.