Gracie Hornsby, Canon Pham, Jennifer Davis, Gary L Darmstadt
Reported data indicated significant behavioral change and suggested that combining education with O&M created a synergistic effect. In contrast, direct observation revealed no significant changes in behavior under any treatment. Stress-testing showed that students frequently reported toilet use when toilets were observed to be locked, and soap use when soap was not observed to be present. Findings from our comparative analysis of WinS literature mirrors the findings of our primary data, revealing that 83% of intervention effects measured with reported data yielded positive results, compared to 58% when measured through observation.
INTRODUCTION: Investments in the provision of water, sanitation, and hygiene (WASH) services in schools (WinS) can only generate health benefits if students use WASH facilities correctly and consistently. Valid and reliable measurement of student WASH behavior is thus critical for evaluating WinS interventions.
METHODS: This study examines the relative validity of direct observation versus reported (survey-based) measures of WASH behaviors in schools, using both primary data and a review of literature. We conducted a sub-analysis of a four-arm, cluster-randomized controlled trial among 200 primary schools in Uttar Pradesh, India, that tested the effect of WASH education and infrastructure operation and maintenance (O&M) on student handwashing and toilet use. We compared unannounced observations of student behavior against self-, peer-, and teacher-reports. We also stress-tested reported data against direct observation of WASH infrastructure and consumables availability. Finally, we conducted a comparative analysis of 42 published WinS studies to contextualize our primary findings.
RESULTS: Reported data indicated significant behavioral change and suggested that combining education with O&M created a synergistic effect. In contrast, direct observation revealed no significant changes in behavior under any treatment. Stress-testing showed that students frequently reported toilet use when toilets were observed to be locked, and soap use when soap was not observed to be present. Findings from our comparative analysis of WinS literature mirrors the findings of our primary data, revealing that 83% of intervention effects measured with reported data yielded positive results, compared to 58% when measured through observation.
DISCUSSION: We find that reported behavioral data systematically overstate the effectiveness of WinS interventions. Current WinS policy and practice is thus grounded in an evidence base that varies significantly depending on the measurement tools employed. Direct observation is less prone to reporting bias but is resource-intensive. Future research should prioritize objective measures of behavior, such as sensors or unannounced checks, to build more robust evidence for WinS policy and practice.