Comparative Survey Design and Implementation: Past, Present, and Future
Wednesday, July 27th
9:00 a.m. - 10:30 a.m.
'What’s Past is Prologue’: The Origin, Development, and Forming of Cross-National Survey Research
Tom W. Smith
Cross-national, survey research emerged out of and developed along with many of the seminal megatrends of the 20th century including globalization and democratization. It was also shaped in important ways by such major historical events as World War II, the advent of post-bellum collective multilateralism, and the spread and collapse of Communism.
The development of cross-national, survey research is an example of what EverettRogers calls the diffusion of innovation. Public opinion polls were created in the United States in the mid-1930s and spread to other countries. Like all diffusions, its development and trajectory was innovation specific and was both aided and hindered by the particular characteristics of survey research itself.
Its expansion was part of the more general process of globalization. Of course in the case of survey research, globalization involved considerable interaction between the global product (survey research) and the local markets and cultures. Thus, as Anthony Heath noted, “Globalization of public opinion polls has not entailed a straightforward spread of a standardized ‘product’ throughout the world in terms of survey conduct.”
Besides being shaped by these overarching megatrends, the development of cross-national, survey research was also influenced by important, historical events. Chief among these were the impact of World War II, the advent of post-war collective multilateralism and the founding of the United Nations, and the emergence of the Cold War and the imposition of the Iron Curtain across Europe.
This paper examines 1) the emergence of cross-national, survey research including the role of early adopters, 2) the stages of expansion from the 1930s to the present, and 3) how the foundational development of comparative research is both shaping and being shaped by contemporary, survey-research methods and practices.
Tom W. Smith is an internationally recognized expert in survey research specializing in the study of societal change and survey methodology. He is Director of the Center for the Study of Politics and Society at NORC at the University of Chicago. Since 1980 he has been a principal investigator of the National Data Program for the Social Sciences and director of its General Social Survey. He is also co-founder and former Secretary General (1997-2003) of the International Social Survey Program (ISSP). The ISSP is the largest cross-national collaboration in the social sciences. Smith has taught at Purdue University, Northwestern University, the University of Chicago, Tel Aviv University, and the University of Cologne.
Why Every Survey is a 3MC Survey
At present a large number of cross-national surveys provide information to measure societal and attitudinal climates, to compare countries, and to follow trends. In the recent past, the strict methodological standards that had long been employed in many national studies tended to be beyond the reach of many comparative studies (Jowell et al., 2007). Nowadays, however, cross-national studies like the European Social Survey (ESS) or PIAAC serve as role models even for national surveys.
The more than 90 thousand registered users of the ESS (March 2016) may not all be aware of the intricacies of designing and implementing a cross-national survey, pursuing the joint aims of high quality and optimal comparability. Ideally, they should be aware of the possible impact of different sampling designs, diverging response rates, varying fieldwork organizations and interviewer staff, the limited relevance of specific concepts within some cultures, and the challenge to field translations that are functionally equivalent in every language and country. Some knowledge on these issues is indispensable, however, when evaluating results and interpreting survey outcomes.
Even though relatively few survey researchers will be actively involved in the design and implementation of cross-national surveys, closer inspection shows that national surveys have quite a number of 3MC characteristics. Obviously, even within a single-country, a survey may have to be fielded in more than one language. This is in evitable in multi-language countries such as Belgium, Canada or Switzerland, but is also the only way to allow minority language or ethnic groups to participate, and thus to achieve a complete inclusive overview of societies. In national survey different response enhancing measures may have to be implemented to reach different socio-economic or cultural groups. Interviewers will have to be trained to obtain the participation of different types of respondents and guide them through the survey questions. And the questionnaire itself should be relevant, accessible and understandable for widely different groups with different skills, competences and interests. Therefore, even though cross-national comparability is not the aim of national surveys, one could state that every survey is essentially a 3MC survey.
Being involved in a 3MC study, or even using 3MC survey data, teaches invaluable lessons to those involved in single country studies. These studies are often based on national traditions, cultural norms, and proven practices. Lessons from cross-national studies can make one aware of national habits in excluding parts of the population, e.g., people living in non-residential households, of clever ways of drawing representative samples, of the risk of involving interviewers in respondent selection, of the effectiveness of respondent incentives and advance letters in different cultures and age groups, of implicit norms in seemingly neutral questions, of factors enhancing the risk of fraud, and of the vagueness of vested questions that turn out to have never been tested sufficiently.
So, experience from 3MC surveys can help survey researchers to understand the 3MC part of national surveys, and to improve national surveys in a variety of ways.
Ineke Stoop is a senior researcher at The Netherlands Institute for Social Research/SCP and Deputy Director Methodological of the European Social Survey. Her key expertise is in cross-national survey methodology with a focus on nonresponse. She is also the chair of the European Statistical Advisory Committee (ESAC).
Facing the Future: Oopportunities and Challenges for Cross-national Surveys: A European Perspective
Most large scale cross-national social surveys in Europe are funded to service academia and policy makers in order to help them address key social, health, economic and environmental challenges. Europe is facing huge challenges in these areas that can be better understood and tackled using data from cross-national surveys: an economic crisis with a young generation in search of jobs, population ageing potentially straining inclusion and innovation, climate change with its pressures to redesign energy, transport and housing patterns, just to name some of the most urgent “Grand Challenges”.
In order to better harness Europe’s cross-national survey resources to address these challenges, a new project – SERISS – has been launched to exploit synergies between leading European infrastructures in the social sciences. The European Social Survey (ESS), the Survey for Health Ageing and Retirement in Europe (SHARE) and the Consortium of European Social Science Data Archives (CESSDA) have come together with the Generations and Gender Programme (GGP), European Values Survey (EVS) and the WageIndicator Survey to help prepare for the future. There is also input from the World Values Survey (WVS), the International Social Survey Programme (ISSP) and the Eurofound family of surveys such as the European Quality of Life Survey (EQLS) to ensure broader discussion and impact.
Whilst the focus for the future of many single nation surveys is on response rates and internet surveys, the demands on cross-national studies are arguably rather different. For instance, using different data collection modes across countries is fraught with difficulty and response patterns differ markedly between countries. Instead the ‘Synergies for Europe's Research Infrastructures in the Social Sciences’ (SERISS) initiative takes a rather broader perspective in terms of preparing for the future. Specific objectives are: to better represent the European population through more coordinated sampling: to strengthen cross-national harmonization across Europe by leveraging recent advances in questionnaire design, translation and coding techniques: and to exploit the advances in software technology for cost-effective web-based interviewing, more efficient fieldwork management and in order to support new ways of collecting data. SERISS also seeks to better connect the world of research-driven social surveys with the world of process-generated administrative and social media data, and to ensure that the ethical and data protection concerns of the respondents are properly taken into account, by creating a consistent and EU-wide framework for all social surveys.
This paper will highlight examples from the project to demonstrate how Europe’s academic social survey and data infrastructures are trying to advance the field and prepare for the future. In that way cross-national social surveys will play a more visible role in our society, allowing further methodological development for the future.
Rory Fitzgerald has been a member of the Core Scientific Team (CST) of the European Social Survey since 2004 and became ESS Director in 2012. The ESS is one of the world’s leading cross-national surveys and has its headquarters at City University London. The ESS has over 80,000 registered users and has led to almost 3000 publications. Prior to working on the ESS he worked at the National Centre for Social Research and the Gallup Organization. In 2016 he was awarded his PhD in Sociology which focused on cross-national survey methodology and in particular the application of the Total Survey Error framework to cross-national surveys.
He plays a leading role in the design, management, and overall coordination of the ESS, and directs the Core Scientific Team. He also works with the national coordinators in each country to ensure the effective implementation of the survey. His key expertise is in cross-national survey methodology, with a focus on questionnaire design, pre-testing, and non-response. He was part of the ESS team that was awarded the Descartes Prize for excellence in scientific collaborative research in 2005. He also played a key role in developing the application for ESS to become a European Research Infrastructure Consortium (ERIC) which was awarded in 2013. In 2016 the ESS was also awarded the status of a landmark infrastructure on the European Strategy Forum for Research Infrastructure roadmap.