- NPI's contribution to IOE
- The value NPI brings
What is Impact Outcome Evaluation?
Since 2015, Nutrition Policy Institute (NPI) has led the California’s Department of Public Health’s CalFresh Healthy Living (CFHL) Impact Outcome Evaluation (IOE) project. Though IOE has undergone periodic design and methodological changes since its inception in 2004, its current design under CFHL’s FFY 2020-2022 Integrated Work Plan is the most rigorous to date.
As part of the three-year design, all Local Health Departments (LHDs) receiving over $400,000 in CFHL funding annually are required to evaluate their school or school-based afterschool interventions with elementary students (4th/5th grade). The number of sites and students to be evaluated was based on power calculations that allow NPI to assess both short and longer-term impacts of CFHL interventions. These full school-year interventions are carried out in the same sites for three consecutive years. Interventions are intended to be robust and multi-component, including direct and indirect education, along with policy, system, and environmental approaches. Evaluation consists of pre- and post-testing students in the fall and spring annually, measuring change in dietary and physical activity behaviors with the Eating and Activity Tool for Students (EATS). This is supplemented with annual completion of the Site-level Assessment Questionnaire (SLAQ), measuring site-level eating and physical activity policies and practices. New in FFY 2020, comparison schools eligible for CFHL services but receiving no CFHL intervention currently or in the recent past were recruited to participate in the three-year evaluation. These schools undertake all of the same evaluation activities at the same time points as intervention schools, but receive compensation at the conclusion of each school year.
- Start-to-finish project coordination
- Study design
- Design of evaluation tools, including the Eating and Activity Tool for Students (EATS) and the Elementary and Out-of-school Time Site-level Assessment Questionnaires (SLAQs)
- Support their use in both paper form and web-based (Qualtrics, Survey123), in English and Spanish
- Institutional Review Board (IRB) preparation, submission, and modification (UC Davis and OSPHD CPHS)
- Comparison site recruitment
- Regular project communication, including development and maintenance of IOE website and monthly newsletters
- Development of evaluation resources:
- Delivery of training and technical assistance:
- Data support, analysis, and dissemination
- Real-time feedback of survey (web-based) progress for LHDs
- Data cleaning and matching
- Data analysis and preparation of IOE findings for USDA Final Report
- Data analysis and preparation of reports and peer-reviewed publications
- Annual development of site-level infographics displaying pre/post EATS and SLAQ data
- Annual development of matched pre/post EATS datasets for each LHD/site
NPI’s leadership and coordination has strengthened CDPH CFHL’s only source of individual-level behavioral impact evaluation data. Successes to date include:
- Recruitment of 36 comparison school sites, including approximately 10 months of cold-calling school districts all over the state
- Development and pilot testing of EATS and accompanying administration protocol
- EATS adopted as youth tool of choice by all CFHL State Implementing Agencies
- Adaptation of EATS (survey and administration protocol) to distance learning and adoption of web-based survey (Qualtrics)
- Cognitive testing of questions added or modified due to COVID-19
- Decreased evaluation burden (data entry) for LHDs using web-based EATS
- FFY 20: Pre-test data collection from 56 intervention and 29 comparison sites (COVID-19 school closures prevented collection of sufficient post-tests)
- FFY 21: Collection of 1,087 matched surveys from 47 intervention sites, and 846 matched surveys from 17 comparison sites
- FFY 21 pre/post findings:
- Frequency of fruit and vegetable consumption improved more in students receiving intervention as compared to those not.
- The adjusted mean difference in change in frequency of consumption of fruit was 0.16 times/day higher in intervention vs. comparison students (p=0.032).
- The adjusted mean difference in change in frequency of consumption of vegetables was 0.45 times/day higher in intervention vs. comparison students (p<0.001).
- 89% of LHD staff surveyed report knowing where to go (always or most of the time) to get IOE questions answered.
- 100% of LHD staff surveyed report receiving a helpful and timely response to IOE-related calls or emails (always or most of the time).
- 84% of LHD staff surveyed report (always or most of the time) attending IOE Quarterly Calls.
- 96% of LHD staff surveyed report they (strongly or somewhat) agree IOE Quarterly Calls provide useful information and updates.