NPI's Role in Impact Outcome Evaluation
What is Impact Outcome Evaluation?
Since 2015, Nutrition Policy Institute (NPI) has led the California Department of Public Health’s CalFresh Healthy Living (CFHL) Impact Outcome Evaluation (IOE) project. Though IOE has undergone periodic design and methodological changes since its inception in 2004, for the first time, as part of CFHL’s FFY 2024-2026 funding cycle, all local health departments (LHDs) conducting minimum intervention (series-based direct education) are required to participate.
As part of the three-year design, LHDs are required to evaluate their school or school-based after school interventions, provided they include at least series-based direct education with grades 4-12. The number of schools and students per school to be evaluated was based on power calculations that allow NPI to assess both local and state-level impacts of CFHL interventions. Acknowledging that school-based interventions carried out by each LHD are highly unique, the current design depends on local variation to determine which intervention dose and/or combination of activities are most effective. We will utilize a full school year evaluation design to ensure we capture the effects of all intervention activities (i.e., direct education, indirect education, and policy, systems, and environmental approaches) in a consistent way state-wide. Evaluation consists of pre- and post-testing students in the fall and spring annually, measuring change in dietary and physical activity behaviors with the Eating and Activity Tool for Students (EATS).
How does Nutrition Policy Institute contribute to Impact Outcome Evaluation?
- Start-to-finish project coordination
- Study design
- Design of evaluation tools, i.e., Eating and Activity Tool for Students (EATS)
- Support its use in both paper form and web-based (Survey123), in English and Spanish
- Institutional Review Board (IRB) preparation, submission, and modification (UC Davis and OSPHD CPHS)
- Regular project communication, including development and maintenance of IOE website and monthly newsletters
- Development of evaluation resources:
- Delivery of training and technical assistance:
- IOE Quarterly Calls in October, January, April, and July
- Brief training modules on relevant aspects of evaluation project
- One-on-one and team support (email, phone, Zoom)
- Annual survey of technical assistance needs
- Data support, analysis, and dissemination
- Real-time feedback of survey (web-based) progress for LHDs
- Data cleaning and matching
- Data analysis and preparation of IOE findings for annual N-PEARS report
- Data analysis and preparation of reports and peer-reviewed publications
- Annual integration of school-level EATS data into NPI data dashboard (coming soon!)
- Annual development of matched pre/post EATS datasets for each school site meeting minimum sample size
What is the value of this contribution?
NPI’s leadership and coordination has strengthened CDPH CFHL’s only source of individual-level behavioral impact evaluation data. Through various IWP cycles, different iterations of this evaluation project, and all of the many challenges (hello COVID!) that come with evaluating school-based CFHL interventions in the real world, it is the LHD-NPI partnership that makes collection of these valuable data possible. By demonstrating the impact and value of these efforts through peer-reviewed publications, annual reporting, and school/local-level dissemination products, we can contribute to the evidence base and further the support for CFHL.