Hero Image

NPI's Role in Impact Outcome Evaluation

What is Impact Outcome Evaluation?

Since 2015, Nutrition Policy Institute (NPI) has led the California’s Department of Public Health’s CalFresh Healthy Living (CFHL) Impact Outcome Evaluation (IOE) project. Though IOE has undergone periodic design and methodological changes since its inception in 2004, its current design under CFHL’s FFY 2020-2022 Integrated Work Plan is the most rigorous to date.

As part of the three-year design, all Local Health Departments (LHDs) receiving over $400,000 in CFHL funding annually are required to evaluate their school or school-based afterschool interventions with elementary students (4th/5th grade). The number of sites and students to be evaluated was based on power calculations that allow NPI to assess both short and longer-term impacts of CFHL interventions. These full school-year interventions are carried out in the same sites for three consecutive years. Interventions are intended to be robust and multi-component, including direct and indirect education, along with policy, system, and environmental approaches. Evaluation consists of pre- and post-testing students in the fall and spring annually, measuring change in dietary and physical activity behaviors with the Eating and Activity Tool for Students (EATS). This is supplemented with annual completion of the Site-level Assessment Questionnaire (SLAQ), measuring site-level eating and physical activity policies and practices. New in FFY 2020, comparison schools eligible for CFHL services but receiving no CFHL intervention currently or in the recent past were recruited to participate in the three-year evaluation. These schools undertake all of the same evaluation activities at the same time points as intervention schools, but receive compensation at the conclusion of each school year.   

How does Nutrition Policy Institute contribute to Impact Outcome Evaluation?

Return to top

What is the value of this contribution?

NPI’s leadership and coordination has strengthened CDPH CFHL’s only source of individual-level behavioral impact evaluation data. Successes to date include:

Evaluation Design, Survey Development & Adaptation (COVID-19)
  • Recruitment of 36 comparison school sites, including approximately 10 months of cold-calling school districts all over the state
  • Development and pilot testing of EATS and accompanying administration protocol
    • EATS adopted as youth tool of choice by all CFHL State Implementing Agencies
  • Adaptation of EATS (survey and administration protocol) to distance learning and adoption of web-based survey (Qualtrics)
    • Cognitive testing of questions added or modified due to COVID-19
    • Decreased evaluation burden (data entry) for LHDs using web-based EATS
Data Collection & Findings
  • FFY 20: Pre-test data collection from 56 intervention and 29 comparison sites (COVID-19 school closures prevented collection of sufficient post-tests)
  • FFY 21: Collection of 1,087 matched surveys from 47 intervention sites, and 846 matched surveys from 17 comparison sites
  • FFY 21 pre/post findings:
    • Frequency of fruit and vegetable consumption improved more in students receiving intervention as compared to those not.
    • The adjusted mean difference in change in frequency of consumption of fruit was 0.16 times/day higher in intervention vs. comparison students (p=0.032).
    • The adjusted mean difference in change in frequency of consumption of vegetables was 0.45 times/day higher in intervention vs. comparison students (p<0.001).
Training & Technical Assistance
  • 89% of LHD staff surveyed report knowing where to go (always or most of the time) to get IOE questions answered.
  • 100% of LHD staff surveyed report receiving a helpful and timely response to IOE-related calls or emails (always or most of the time).
  • 84% of LHD staff surveyed report (always or most of the time) attending IOE Quarterly Calls.
  • 96% of LHD staff surveyed report they (strongly or somewhat) agree IOE Quarterly Calls provide useful information and updates.