Assessing and Modelling Unit Nonresponse Bias in Probability-Based Web Surveys

BACKGROUND. Web surveys have become the predominant survey mode, especially when combined with nonprobability sampling. However, this proposal addresses academic, government and other social science research that relies on probability surveys using traditional survey modes (mail, telephone, face-to-face), which are now transitioning to probability-based web surveys (PBWS). In the context of total survey error, web surveys are already known for their strong measurement properties, while noncoverage is a diminishing issue, but nonresponse remains a major problem for PBWS. Recruitment into PBWS relies on traditional survey modes, as there are no email lists for general population surveys, which are the primary domain of this proposal. However, to save costs, recruitment into PBWS is mostly carried out by mail (or email where possible), which significantly reduces response rates, which are often even below 20%. This is very low compared to traditional surveys, where response rates of 70% or 80% were once the standards that have been gradually abandoned (with little justification). Such low response rates increase the danger of nonresponse bias, which means that the estimates from PBWS can differ notably from the true values of the parameters. This presents a serious threat to the quality of empirical social science research. RESEARCH GAP. Previous evaluations of PBWS have focused on sample representativeness in terms of control variables (e.g., R-indicator) or measurement equivalence. No systematic question-level evaluations have examined PBWS-specific nonresponse biases that may occur for some topics (e.g., poverty) but not for others (e.g., elections). We lack a thorough insight into the moderators associated with this bias. OBJECTIVES: The main objective is to comprehensively elaborate and model the PBWS-specific nonresponse bias. The overarching research question is: Under what circumstances can PBWS be used instead of traditional surveys, given the impact of nonresponse bias? The sub-objectives are: (1)  Analyse and model the moderators of PBWS-specific nonresponse bias, particularly response rate but also research topic, respondent characteristics and methodological features; (2)  Predict PBWS-specific nonresponse bias for any question; (3)  Evaluate PBWS-specific nonresponse bias in the contexts of accuracy and cost. METHOD. Objective (1) relies on a systematic literature review and secondary data analysis. PBWS-specific nonresponse bias will be investigated at the question level for selected academic and official surveys in Slovenia and for the CRONOS European Social Survey (ESS). A separate experimental survey will be conducted in parallel with ESS 2025. Objective (2) will draw on the results of the first objective to develop a predictive model using advanced statistical and knowledge discovery methods. In addition, the model will be integrated as a prototype module in an open-source web survey tool. Objective (3) will simultaneously observe the costs and accuracy (which is a measure that integrates bias and variance). IMPACT. The results will provide: A comprehensive elaboration of nonresponse bias in PBWS; Decision-making guidance on the suitability of PBWS for different objectives in social science research; A feasibility study for a stand-alone tool to predict bias in PBWS, complementing the Survey Quality Predictor (SQP), which predicts measurement error; Contributions to the transition of ESS 2025 to PBWS; Facilitation of the implementation of PBWS in Slovenian official and academic surveys. TEAM. The principal investigator has demonstrated excellence by publishing in high-impact journals. The research team has pioneered web survey research, authored highly cited articles, written a seminal textbook on web survey methodology and developed an open-source web survey tool (1KA). The advisory board consists of leading international scholars in the field. This promises the successful achievement of the project’s ambitious objectives.