The arduous task of assessing impact

Return to UC Cooperative Extension | Agricultural Experiment Station

The California Naturalist program seeks to evaulate the impact of its efforts to train volunteers in nature conservation and stewardship.

One of the common challenges across all environmental work is determining the impact we are making. Are we making a difference? Is it at the scale that we hoped for? Are people changing their attitudes or actions as a result of our programs and outreach materials? Has the environment benefited as a result of our effort? Are there areas that we could better spend our limited time and resources?

Unlike farmers, for example, we can't report back changes in yield per acre planted or larger fruits as a result of our efforts. We can't always prove whether an end result is correlated or caused by our efforts. But we still have to find a way to link our efforts to outcomes and accurately quantify some of the qualitative information about attitudes, behavior, knowledge, skills, and impacts. Evaluation is typically an important component of demonstrating a program's accountability and continued worth to funding sources. There are resources to ensure your environmental program is using the best available peer-reviewed protocol and methods.

As a new ANR program, the UC California Naturalist Program constantly seeks ways to evaluate program impacts. Working with UC Davis School of Education's Heidi Ballard at the helm of the evaluation process, we received a two-year NSF Pathways grant to develop and test survey instruments to evaluate the impact of the California Naturalist program. We conducted interviews of class participants in 2012 and 2013 and included questions about the course, perspectives on science and the environment, prior knowledge of the content delivered during the course, information about each participant, and finally changes in participant's knowledge and scientific and environmental perspectives. We worked with a committee of experts that included academic experts in the field of citizen science from Cornell and other universities.

The results of this two-year study will help direct the future course of our program. We found that overall course satisfaction was high with a majority of participants selecting the highest level of satisfaction, four individuals selecting somewhat satisfied and none selecting anything below. Instructors all received very good or excellent rankings. Ninety-six percent of those surveyed responded that they do plan to volunteer, which indicates that the participants who are receiving this training will be applying it to land stewardship, interpretation, citizen science and other activities that help nature and people. The mission of our program is to foster a diverse community of naturalists and promote stewardship of California's natural resources through education and service. Our findings support our program tremendously, and also point to opportunities for growth, like making sure the diversity of program participants better reflects the diversity in California.

Evaluation Question: What aspects of the class were most beneficial to learning about science and Naturalist skills?

Evaluation is a science with logical steps that require expertise and resources to execute in a meaningful way. A blog post detailing all the ways a program can tackle an evaluation plan would be redundant at best, and misleading at worst. I am not a social scientist, nor an expert in evaluation, but I have received an inordinate amount of help in program evaluation from internal training at ANR, university colleagues, and a few online resources highlighted below. There are some excellent free resources available that cover both the pedagogy and practical steps in creating an evaluation plan.

UCANR ToolKit for Assessing IPM Outcomes and Impacts While specifically geared toward an IPM program, this will be useful for anyone needing a refresher or solid introduction, steps, and links to peer-reviewed literature on evaluation planning.
The NSF User-Friendly Handbook for Project Evaluation (2012) Aimed at people who need to learn more about both the value of evaluation and how to design and carry out an evaluation, rather than those who already have a solid base of experience in the field. It builds on firmly established principles, blending technical knowledge and common sense and presenting both quantitative and qualitative evaluation methods.
Center for the Advancement of Informal Science Education (CAISE) The resources on this NSF-funded website are designed to assist project directors, and evaluators, by referring to reports on past projects to build on prior work and to key resources to learn about integrating and managing evaluation.
The American Evaluation Association's public e-library Best if you have a search term in mind or time to head down various rabbit holes!