Full Day Courses - Monday, July 25th 9:00 a.m. - 5:00 p.m.
Half Day Course - Tuesday, July 26th, 9:00 a.m. - 1:00 p.m.
Design and Implementation Challenges of International Surveys
Length: Half Day
Instructor: Zeina Mneimneh
Zeina Mneimneh is an assistant research scientist and a survey methodologist in the Survey Research Center, University of Michigan. She has an MSc and a PhD in Survey Methodology from University of Michigan. She has more than 15 years of experience in designing, conducting, and overseeing surveys in international settings and is a collaborator in the cross-national World Mental Health survey initiative. Her main operational interests include conducting surveys in areas affected by conflict, cross-cultural survey design and implementation, and international capacity building. Her substantive methodological interests include interview privacy, social desirability effects, and adaptive measurement design.
Description: This short course focuses on the challenges faced in the design and implementation of face-to-face international surveys. Issues related to contracting survey organizations, sample design and sample management, questionnaire development and testing, response processes, interviewer recruitment and training, field operations, and quality monitoring will be discussed with examples drawn from surveys conducted in Africa, the Middle East, and Southeast Asia. The course will also cover cultural dimensions and how these interact with the survey process and survey error. The course will include in-class exercises mainly presented through case studies. Participants who are working in the field of survey research in international context are highly encouraged to share and present their experiences and any challenges they are facing.
Mixed Mode Methods in the 3MC Context
Length: Full Day
Instructors: Edith de Leeuw and Tuba Z Suzer Gurtekin
Dr. Edith de Leeuw is a full professor in survey methodology and statistics at the University of Utrecht. She is an associate editor for the Journal of Official Statistics (JOS) and editor of the international journal Methods, Data, Analysis (MDA) and is on the editorial boards for leading journals in the field of survey methods. Edith organized and taught international workshops and seminars on mixed-mode, meta-analysis, nonresponse, computer assisted data collection, questionnaire design and data quality in surveys and was the editor of 3 internationally renowned books on survey methodology (The International Handbook of survey methodology, Advances in telephone methodology, and Survey Measurement and Process Quality). At present her research focuses on Total Survey Error and mixed-mode data collection.
Dr. Tuba Suzer Gurtekin is a research fellow in survey methodology at the University of Michigan. Her research and practice on mixed-mode surveys include design and analysis of mixed-mode customer satisfaction studies, design and analysis of mixed-mode survey experiments in an ongoing monthly general population telephone survey and evaluation of mixed-mode survey inference methods. She taught classes on fundamentals of survey methodology, randomized and nonrandomized design, data collection and analysis practicum. Her current research focuses on mixed-mode survey inference methods, response style adjustment methods and respondent driven sampling data analysis.
Description: Mixing survey modes appears almost inevitable today. Three important reasons to use a mixed-mode survey design are improving coverage, increasing response rates and reducing costs. However, there are also potential drawbacks, such as, increased administrative and logistic burden, and potential for mode specific measurement error.
In the first part of the course, we address the major variants of mixed-mode data collection designs, issues in mixed-mode and mixed-device questionnaire design, and management of mixed-mode projects. In the second part, we discuss issues in the analysis of mixed-mode surveys, going from an introduction to more advanced statistical techniques.
We discuss several mixed-mode designs for cross-sectional, longitudinal, and cross-national surveys. We summarize the empirical evidence for reducing coverage and nonresponse error and then focus on measurement error. We review three related issues in mixed-mode survey practice: design, diagnosis, and adjustment. The first step in a mixed mode survey should be to design the survey in such a way that it minimizes mode measurement effects. We will give empirical examples. In the second step, one has to investigate to what extent apparent differences between modes are the result of intended differential selection of respondents to different modes, and hence help to reduce coverage error. This is followed by estimating the (unwanted) mode measurement effects while controlling for the selection effects. If there are unwanted mode effects in the measurements, adjustments are needed in the analysis phase. We illustrate this process with examples.
Sampling in the 3MC Context
Instructors: Stephanie Eckman and Colm O'Muircheartaigh
Length: Full Day
Stephanie Eckman is senior methodologist at Research Triangle International. She has also consulted on surveys with the World Bank.
Colm O'Muircheartaigh is Professor at the Harris School and Public Policy at the University of Chicago and a Senior Fellow at the National Opinion Research Center. For many years he worked as a sampler for the World Fertility Survey.
Description: This course will cover methods of selecting complex samples. The course will begin with simple random sampling and move onto stratified, clustered and multi-stage sampling. For each method, students will learn the relevant formulas for point estimates and variance estimates; however, the course will emphasize application over theoretical proofs of the formulas. Students will learn the benefits and costs of the different sampling methods and when each is appropriate. Because 3MC surveys often involve working in countries where population registers are not available and census data is not up to date, we will also discuss alternatives to classical sampling techniques that may be more useful in such contexts.
Questionnaire Design for 3MC Studies
Length: Full Day
Instructors: Ting Yan and Sunghee Lee
Ting Yan is a Senior Survey Methodologist with Westat. She obtained her Ph.D. in survey methodology from Joint Program in Survey Methodology, University of Maryland. She has more than 15 years' experience working in survey organizations and has been teaching Questionnaire Design since 2013 to students in both University of Michigan and University of Maryland.
Sunghee Lee is an Assistant Research Scientist at the Institute for Social Research, University of Michigan. She hold a Ph.D. in survey methodology from Joint Program in Survey Methodology, University of Maryland. Her work has been focused on methodological issues in data collection with minority populations, in particular, integrating cultural norms affecting cognition to survey measurements.
Description: This class consists of two modules: 1. Questionnaire design; 2. Considerations for 3M survey questionnaire design.
The first module provides an overview of questionnaire design issues, including research questions to questionnaires, survey response model, asking various types of questions (factual, quasi-factual, attitudinal, and sensitive questions), response scales, mode effects, tools for testing questionnaires and considerations for 3MC surveys.
The second module delves into 3MC surveys extensively by discussing questionnaire translation, the definition of culture, the role of cultural norms and cognition for 3MC surveys, measurement comparability issues with respect to reporting heterogeneity, methods to address measurement noncomparability, cultural adaptations as well as questionnaire evaluation for 3MC surveys.
Cognitive Interviewing Methodology in 3MC Contexts
Length: Half Day
Instructors: Kristin Miller and Meredith Massey
Kristen Miller, Ph.D directs the Question Design Research Laboratory within the National Center for Health Statistics (NCHS), CDC. Her writings have focused on question comparability, including question design and equivalence for lower SES respondents and the improvement of evaluation methods for cross-cultural and cross-national testing studies. She is a co-editor of two survey methodology books: Cognitive Interviewing Methodology (2014) and Question Evaluation Methods (2011). Through her tenure at NCHS, she has led collaborative international testing projects with statistical agencies and organizations including the European Social Survey, the World Bank, the World Health Organization and the United Nations. Dr. Miller holds a PhD in Sociology from the University of Delaware.
Meredith Massey, Ph.D is a behavioral scientist at the Question Design Research Laboratory within the National Center for Health Statistics (NCHS), CDC. She has worked on projects focusing on child disability (UNICEF), adults with chronic healthcare needs and high impact chronic pain. Dr. Massey holds a PhD in Public Health from Johns Hopkins School of Public Health.
Description: This course will cover three topics: 1) question response theory, specifically, the influence of socio-cultural context on cognitive processes for question response, 2) conducting cognitive interviewing studies in 3MC settings, and 3) software applications and tools to assist in cognitive interviewing studies.
Survey Translation Methods for 3MC Studies
Length: Full Day
Instructors: Dorothée Behr, Brita Dorer and Alisú Schoua-Glusberg
Dr. Dorothée Behr has been a cross-cultural survey methodologist at GESIS – Leibniz Institute for the Social Sciences, Mannheim, since 2006. She holds a degree in translation studies and a doctorate on questionnaire translation. From 2015-2016, she served as an interim professor for Applied Translation Studies at Magdeburg-Stendal University of Applied Sciences. Her research and services (consultancy, training) focus on all aspects related to questionnaire translation in cross-cultural surveys. Furthermore, she has been involved in research that pioneers cross-cultural web probing to identify item comparability. Previous projects include the European Social Survey, the Programme for the International Assessment of Adult Competencies and the Programme for International Student Assessment.
Brita Dorer is a researcher at GESIS – Leibniz Institute for the Social Sciences, specialized in the field of questionnaire translation and adaptation. She is heading the translation team of the European Social Survey (ESS) and leading the Workpackage on “Maximising equivalence through translation” in the EU-funded SERISS cluster project. Her current scientific interests include the quality of questionnaire translations and adaptations, translation and assessment methods, translatability of source questionnaires / advance translations, intercultural aspects of questionnaire translation and translation process research. She is currently preparing a PhD on advance translations carried out for improving the translatability of survey questionnaires in the ESS. She holds a degree in English, French and Italian translation studies from Johannes-Gutenberg-University Mainz, FTSK Germersheim. She worked for many years as a translator in different fields and had teaching assignments at the Universities of Strasbourg, Karlsruhe and Mainz/Germersheim. She has been involved in translating survey questionnaires into German, such as ESS, ISSP, PIAAC and SHARE.
Alisú Schoua-Glusberg is a cultural/linguistic anthropologist (Northwestern University Ph.D., 1985) and survey methodologist. She has worked in survey operations since 1984, first at NORC at the University of Chicago where she was Director of the Survey Operations Center and Translations' Coordinator. She has directed large-scale, longitudinal surveys at NORC, at Harvard Medical School (Director of Survey Operations for the Project on Human Development in Chicago Neighborhoods), Metro Chicago Information Center and IMPAQ International. She founded Research Support Services in 1996, which provides qualitative and quantitative research services, as well as instrument translation. She has presented and trained extensively on survey translation, including co-teaching an AAPOR survey translation short course in 2004 and webinar in 2014. She was a member of the US Census Bureau Expert Panel on translation and is a member of the European Social Survey Translation Expert Group. She is a member of the U.S. Census Bureau National Advisory Committee on Racial, Ethnic and Other Populations. She has organized and managed translation efforts for numerous U.S. federal and academic surveys for over 25 years.
Description: This short course tackles questionnaire translation from different angles. First, participants will learn about the interplay between source questionnaire quality and translation quality. The focus will be on adequate processes to develop a source questionnaire and on aspects to consider in terms of its cross-cultural implementation. Second, do’s and don’ts in questionnaire translation itself will be covered. Third, translation and assessment procedures will be presented; these will cover in particular the team approach, including needed personnel, and pretesting of translations. Forth, additional management issues will be discussed, such as harmonization between language versions, communication between those who commission translations/project managers and translators, documentation for different target groups, and tools. Finally, time will be set aside for participants’ questions, projects or challenges.
Sources of Error in Cross-Cultural Surveys
Length: Half Day
Instructor: Emilia Peytcheva
Emilia Peytcheva, is a research survey methodologist with RTI International and an adjunct assistant professor at the University of North Carolina- Chapel Hill. She holds a PhD. in survey methodology from the University of Michigan. Dr. Peytcheva's research expertise includes measurement error-inducing factors in cross-cultural research and the interplay among survey errors and their combined effect on total survey error. Her interests include methods for minimizing measurement error induced by the survey questionnaire.
Description: The course will focus on the different sources of measurement error in cross-cultural surveys (respondent, language, survey instrument, interviewer, and mode). It will begin with a theoretical background for cross-cultural differences drawing on cross-cultural psychology; present relevant psycholinguistic theories to demonstrate how language can influence each stage of the response formation process, and discuss issues related to bilingualism. When the survey instrument is discussed as a source of measurement error, special attention will be given to issues related to translation.
The course will briefly touch on other sources of error (sampling, coverage, nonresponse, and processing) and will conclude with a case study on examining the effect of language on survey responses.
Detecting and Interpreting Item Bias in 3MC Studies
Length: Half Day
Instructor: Jose-Luis Padilla
Jose-Luis Padilla received his Bachelor’s and Master’s degrees in Psychology from the University of Granada (Spain). He is an Associate Professor at the Department of Methodology of Behavioral Sciences at the University of Granada (Spain). He teaches graduate and master courses of psychometrics, questionnaire design, and cognitive pre-test methods. His current research focuses on psychometrics, validity, and cross-cultural research within a mixed method framework combining quantitative and qualitative methods. He has been editor and co-author of 2014 book Cognitive Interviewing Methodology (John Willey and Sons, Inc. USA).
Jose-Luis Padilla has been commissioned by the Spanish National Statistical Institute to pre-test survey questionnaires of national and international surveys, among others, the Spanish Census 2011, European Interview Health Survey, European Module of Disability, Spanish Health Survey, and the Spanish Labor Survey. He is also a member of the QUEST (Questionnaire Standards for question evaluation methods) network, serving on the organizing committee since 2010, and the organizing and publishing committees for the 2016 QDET2 Conference.
Description: The growing use of psychological and health scales in cross-national educational, public opinion, and quality of life surveys makes necessary to address how Differential Item Functioning (DIF) can undermine validity of comparative interpretations based on survey data. The main aim of the course is to present a practical, comprehensive approach to detecting and interpreting DIF in scales included in multi-national survey questionnaires. Within a mixed-methods research framework, the course will present widely used DIF statistical techniques (Mantel-Haenszel, Logistic Regression and Differential Step Functioning), for detecting DIF in polytomous items, and qualitative methods like expert appraisal and cognitive interviewing to interpret DIF results. Practical examples of DIF analysis and interpretations will be developed using data bases of international surveys like PISA, European Social Survey, and SHARE projects. Finally, the general structure to build validity arguments of the equivalence level reached for comparative interpretations using DIF results will be taught.
Paradata in 3MC Studies
Length: Half Day
Instructor: Frauke Kreuter
Frauke Kreuter is currently the director of the Joint Program in Survey Methodology at the University of Maryland, USA; Professor of Statistics and Methodology at the University of Mannheim; and head of the Statistical Methods Research Department at the Institute for Employment Research (IAB) in Nürnberg, Germany. She received her Master in Sociology from the University of Mannheim, Germany and her PhD in Survey Methodology from the University of Konstanz. Before joining the University of Maryland she held a postdoc at the UCLA Statistics Department. During her term as AAPOR standards chair she chaired the AAPOR task force report on Big Data. Frauke Kreuter is Fellow of the American Statistical Association. She published numerous journal articles, textbooks and edited volumes, among others the Wiley book on “Improving Surveys with Paradata”, which forms the basis of this course and is part of the course package.
Description: Paradata arise in the process of survey data collection. Their collection is often not designed by the researcher, instead they arise as a byproduct in computer aided production. With this they share many features that are attributed to Big Data: they are available at a high velocity, come in a variety of formats, at a large volume, and often lack veracity. However their appeal is the ubiquitous availability in many cross-cultural and cross-national surveys. This short course will present an overall framework, highlight examples of the utility of paradata, and discuss challenges associated with their use.
3MC Analysis: Multilevel SEM
Length: Full Day
Instructor: Bart Meuleman
Description: Social scientists are regularly confronted with data that has a hierarchical structure: Citizens are nested within countries (or regions), pupils within schools, and employees within companies. More often than not, research questions are related to the interplay between these different levels (e.g. how contexts influence individual behavior or opinions). Multilevel SEM (MLSEM) takes this clustering into account and simultaneously allows controlling for measurement error by introducing latent variables. This course gives an overview of MLSEM techniques from the perspective of the applied researcher. It covers several models that are particularly useful in cross-cultural and cross-national research, such as multilevel confirmatory factor analysis (including the evaluation of measurement equivalence), multilevel full structural equation modeling and multilevel mediation analysis. The course employs examples from comparative value and attitude research, using two-level data from the European Social Survey. The software packages Mplus is used for all examples and during lab exercises. Detailed discussion of the command syntax and the interpretation of the output are given.
Participants should take home the following issues:
- An understanding of the conceptual foundations and basic formulation of the multilevel SEM model.
- The ability to understand, interpret and explain the output from Mplus.
- An appreciation of the advantages and disadvantages of multilevel SEM as compared with multilevel regression and multigroup SEM using nested data.
- Strategies for developing and testing multilevel models
- How to use of multilevel SEM to explain measurement non-invariance in survey data