A function of Cooperative Extension is to develop quality curriculum and outreach programs to communities around the nation. However, many Extension staff have found it challenging to effectively deliver these to the community, including underserved audiences (Davis et al. 1990). One venue for program delivery to school-age children is through school-based afterschool programs, which are well-established throughout the country. Also called Expanded Learning, these programs take advantage of the often-underutilized hours after the school day to enhance student learning. Afterschool programs are free from the constraints of mandated standards, allowing for rich modes of program delivery (Pelcher and Rajan 2016). Because they are flexible, and those who administer them are generally eager to include content-rich learning experiences, afterschool programs are well-suited to host science education programs that reach beyond the scope of the school day (Chi et al. 2008). Despite these advantages, afterschool staff have often found it challenging to deliver science programs effectively.

Samuel Sandoval Solis, Associate Professor and Cooperative Extension Specialist in Water Resources at UC ANR, shares his groundwater model. Photo: Marianne Bird.

State and federal funding support afterschool programs in economically disadvantaged communities. Goals for these programs include providing academic support and enrichment to complement what children learn during school hours (Deke et al. 2012). This support and enrichment are especially critical in providing quality science, technology, engineering and math (STEM) programming. As of 2015, only 38% of fourth grade students in the United States scored above average or proficient in science (NAEP 2015). In California, 55% of fifth graders scored proficient in science in 2015 (CDE 2015). The gap in scientific literacy is especially prominent for youth of color (NCES 2018).

Afterschool programs face several challenges. Instructors in afterschool programs are not typically credentialed teachers (Chi et al. 2008). As well, afterschool staff, particularly in low-income communities, often have little background in subject expertise or in training as teachers, especially in science pedagogy, such as inquiry-based exploration (Freeman et al. 2009). Increasing staff competence in understanding content and raising staff confidence in facilitating science learning are important components in providing quality STEM programming (Harlen 1997; Harlen and Holroyd 2007; Miller 2005).

Afterschool programs benefit from science programming that can meet the dual purpose of increasing students' science knowledge while enhancing the staff's ability to effectively deliver science education to students (Smith and Schmitt-McQuitty 2013). To help meet this dual goal, the University of California 4-H Youth Development Program created 4-H Water Wizards in 2006. This project provides water education to children in after school programs, with the longterm goal of fostering a citizenry that understands and can make informed decisions about water. The program also serves to increase afterschool staff competencies in teaching science. It incorporates best practices in STEM education for afterschool settings, including the Experiential Learning Model, staff training strategies and service-learning (see Ripberger and Blalock 2011). We undertook a multi-year study, described here, to determine if the program is effective in meeting its two-pronged goal. We share our lessons learned from implementing this project especially with regard to working with afterschool audiences.

4-H Water Wizards encourages youth to investigate water through models, experiments, and surveys. Here, a student shares the results of seeds placed on a paper towel dampened with fresh water, which they will compare to a salt water sample. Photo: Marianne Bird.

4-H Water Wizards project

Water has been called California's “new gold,” a precious and limited natural resource that is critical to agriculture, communities and the environment. Water is a driver of California's large agricultural economy and is often the center of policy debate. The issues surrounding water are also personal ones; individuals and families increasingly face issues of water conservation and clean waterways (Gregory and Di Leo 2006). Education is the first step in helping children understand and appreciate water and its role in their communities (Gregory and Di Leo 2006). The 4-H Water Wizards project helps foster this understanding and appreciation through three components: community partnerships, curriculum and staff training.

Community partnerships

Sacramento County University of California Cooperative Extension (UCCE) partnered with the City of Sacramento's afterschool program to deliver 4-H Water Wizards. During the years of our study, 2012–2016, 20 afterschool sites participated in the program annually (10 sites in the fall and 10 in the spring) and included a total of 2,039 students in grades four through six. At least 50% of students in participating schools received free or reduced-fee lunches and almost 85% were students of color (table 1). The 143 program staff members leading the project were equally diverse (table 2).

TABLE 1. Ethnicity and gender for total youth enrolled in the 4-H Water Wizards Project, 2012–2016

TABLE 2. Ethnicity and gender for total afterschool program staff who delivered the 4-H Water Wizards Project, 2012–2016 (duplicates included)

Curriculum

The 4-H Water Wizards curriculum (table 3) consists of 11 weekly sessions, each 45 to 60 minutes long, and encompasses three units: Water and the Environment (the water cycle, watersheds, water pollution and conservation); Water Properties (taste test, salinity, density and hardness); and a Service Learning Project (exploring service learning, planning and delivering a water-related community service project, and project evaluation). The curriculum utilizes guided inquiry, which encourages students to construct knowledge through investigation. In the project, students engage in hands-on activities that promote exploration, such as experimenting and using models. Encouraging students to construct knowledge is an established science education technique rooted in education philosophy, and it guides contemporary learning practices (Dewey 1916; Russ and Berland 2019). In addition to hands-on activities, students at each afterschool site attended an afternoon field trip to the nearby American River Water Education Center, where they visited the Folsom Lake watershed; learned about the Folsom Dam and its role in water storage and flood control; and discussed the need for water to support agriculture, people and the environment.

TABLE 3. Components of the 4-H Water Wizards project

4-H Water Wizards participants gather around an Enviroscape model where afterschool staff facilitate a lesson on non-point source pollution. Students add elements representing car oil, fertilizer and pesticides to the model and note what happens when runoff from rain carries pollutants into waterways such as rivers and lakes. Photo: Marianne Bird.

Staff training

4-H Water Wizard developers designed staff training to include best practices for science education in urban settings (Ripberger and Blalock 2011). Afterschool program staff attend three, three-hour evening trainings, one before each unit (a total of nine hours). The trainings included an introduction to teaching inquiry-based science, a modeled lesson, the opportunity to practice delivering lessons to each other, and reflections on the material and their experience in delivering the project. This type of incremental training over an extended period of time (as opposed to an episodic workshop) has been shown to be an effective professional development strategy (Smith and Schmitt-McQuitty 2013). Program staff left each session with all the materials they needed to teach the unit for which they had just been trained. 4-H staff were available for additional support, including site visits.

Evaluation of 4-H Water Wizards

Our research question was twofold: Does the 4-H Water Wizards project significantly impact staff confidence in teaching water-related science material? And, does the project increase student knowledge about water and lead to conservation practices? We designed and conducted our multi-year study to assess the project's outcomes. Specifically, we wished to determine if

• afterschool program leaders became more confident leading science activities as a result of delivering 4-H Water Wizards,

• the 4-H Water Wizards project was effective in raising children's knowledge about water and awareness of water issues,

• youth participating in the project changed their behavior related to water usage as a result of participating in the project and

• youth and program leaders enjoyed the project.

We used a one-group, pretest-posttest study design to help us answer these questions (Creswell and Creswell 2017). For four consecutive years, from 2012 through 2016, we collected data from 59 staff members and 469 students from a sample of 21 afterschool sites that implemented 4-H Water Wizards curriculum and opted to be a part of the study. Many sites participated for multiple years.

We present our evaluation as two parts, focusing on staff evaluation and outcomes in the first and student evaluation and outcomes in the second. Following this we discuss our findings and lessons learned from this process.

I. Evaluating staff knowledge and confidence

Sample. The 59 afterschool staff who participated in our study were ethnically diverse, reflecting the diversity of the students, and 78% female. The amount of time that each staff member worked in the afterschool program varied (mean = 2.9 years; range = 1 month to 14 years).

Data collection. We designed two surveys to assess changes in staff knowledge about, and attitudes toward teaching science. The first was a pre-post-questionnaire that assessed changes in the staff's perceived experience, ability and comfort in teaching science. We administered the pre-questionnaire at the first training session, and the post-questionnaire after the final training session — to measure change. In the questionnaires, we asked staff to rate themselves on their experience and understanding of science, comfort level in teaching and their enjoyment of science using a four-point Likert scale (1 = poor to 4 = very good). The questionnaires also contained open-ended questions about why staff were interested in the project (on the pre-questionnaire), what they learned in teaching the project (on the post-questionnaire) and what they considered “important things to do” when teaching science (on both pre- and post-questionnaires).

The second staff survey was an end-of-project evaluation that included a retrospective self-report on knowledge of water information and teaching science. We asked Likert-scale questions concerning training, confidence and enjoyment of the project, and we included open-ended questions to assess experience in delivering the curriculum. The evaluation also included a program checklist where staff indicated which curricular components they completed with their students.

Quantitative analysis. We matched pre- and post-questionnaire data, compiled it in Excel and analyzed it in SPSS (IBM, N.Y.) with paired sample t-tests to check for significant differences in scores. We did the same for the quantitative data we collected through the end-of-project staff evaluations.

Qualitative analysis. We analyzed the short-answer qualitative data for insights into the program staff's experience with, and insights about, the project. We compiled and summarized the responses related to program experience using a grounded-theory framework (Strauss 1987), noting emergent themes. We quantified the qualitative data to understand theme prevalence and changes in staff understanding of important factors in teaching science. Staff provided a checklist of project activities they completed with students, and overall reports showed that, on average, they completed more than 90% of the activities.

Staff outcomes

Growth in competence and confidence. Afterschool staff rated themselves significantly higher (P = 0.05) on the posttest questionnaire than they did in the pretest questionnaire in the following areas: Experience teaching science to children, understanding of science, comfort level teaching about water and the environment, and enjoyment of science (fig. 1, table 4). In the end-of-project evaluation retrospective survey (fig. 2), afterschool staff reported increased understanding in both content knowledge (the water cycle, water properties and water issues) and teaching methodology (how to teach science and the importance of inquiry in that process).

FIG. 1. Afterschool program staff pre-program and post-program scores from the questionnaires rating (1 = poor to 4 = very good) about various skills and attitudes in teaching science (n = 59).

TABLE 4. Afterschool program staff pre- and post-program scores rating themselves (1 = poor to 4 = very good) on various skills and attitudes in teaching science (n = 59)

FIG. 2. Afterschool program staff retrospective assessment rating their understanding about water and teaching science prior to and after delivering the project.

The end-of-project evaluation demonstrated two main themes related to what staff had learned: content knowledge (water properties, the water cycle, the value of water and the importance of conservation) and teaching pedagogy (the importance of inquiry and teaching science through learner-centered methods). Staff comments included:

I learned more about water and the water cycle. I also learned how to teach and learn about water in a fun and engaging manner.

… how valuable water is.

I built on my vocabulary and great experiments. Also after the field trip I realized I waste too much water.

A community engagement project caps 4-H Water Wizards. Here, a program leader helps youth explore what a community is and what it means to volunteer as they prepare to plan their project. Photo: Marianne Bird.

As a result of delivering the program, staff members' thoughts shifted regarding what was important when teaching science. Whereas “fun/student engagement” and “being prepared” were typically cited as important in the pre-questionnaire, a greater percentage of staff said in the post-questionnaire and end-of-project evaluation that “encouraging exploration” was important (22% pre-questionnaire to 29% post-questionnaire). This was the most cited response by the end of the program, as expressed in the following comments.

I learned that the children learn more when they're actually doing the project instead of just listening to it.

There is nothing wrong with not knowing an answer in Water Wizards and science in general.

I learned how to be open in questions and the answers so the kids explore further in their thinking instead of cutting their thought off with my own answer.

Engagement and reflection. Almost half (46%) of the afterschool staff cited the curriculum content or learning about water as the reason they chose to participate in 4-H Water Wizards. Most staff (80%) indicated that they would choose to deliver 4-H Water Wizards again, and the remaining 20% said maybe they would. Typical comments included:

Absolutely, important topic, engaging, materials provided.

Yes & No [to doing the project again]. The [training] meetings are inconvenient but I love doing the projects.

Staff cited both “activities” and “hands-on learning” as the best part of the program (38%), followed by the field trip (14%).

Children were engaged in every activity, also were very excited to do hands-on activities.

Hands on activities. Some projects felt like very big undertakings that were better in theory than in practice.

I very much enjoyed everything I did with the children, especially the field trip.

Almost one-third of staff (29%) identified time as the greatest challenge in delivering the project regularly. Other challenges included “holidays and professional development days,” “homework and other activities” and “… pacing myself so I could get to an activity a week.” Fourteen percent said they had challenges with specific activities, while 24% reported no challenges in delivering the project. When asked what would strengthen the project, some staff mentioned changing the time of the training sessions from evenings (after a long program day) or providing more training.

II. Evaluating student knowledge, behavior change and enjoyment

Sample. Our student sample consisted of 496 ethnically diverse elementary school students in fourth through sixth grades from 21 afterschool sites.

Data collection. We designed pretest and posttest questionnaires, based on key curricular concepts, to assess changes in student knowledge and attitudes towards water use and conservation. Both questionnaires included nine multiple-choice and short-answer questions to assess knowledge. An example of a question that appeared on both questionnaires was:

The part of the water cycle where water returns to the air is called

a. waterfall
b. evaporation
c. observation
d. condensation

The posttest questionnaire also contained additional questions to gauge behavior change regarding water conservation. We used these questions, such as the one following, to assess whether students enjoyed 4-H Water Wizards and if they were using less water as a result of participating in the project.

Are you using less water since participating in 4-H Water Wizards? Yes/No

If so, what are you doing to conserve water?

Children received the pretest questionnaire in their afterschool program within the first two weeks of the project. They filled out the follow-up questionnaire at the conclusion of the project. 4-H staff administered the assessments.

Analysis. As we did with the staff data, we matched pretest and posttest quantitative data, compiled it in Excel and analyzed it in SPSS. We conducted paired sample t-tests to assess whether the project impacted students' knowledge about, and attitudes toward, water.

Student outcomes

Increased knowledge about water science. Most student participants gained knowledge about water through the 4-H Water Wizards project. Mean posttest scores were significantly higher than pretest scores for all four years combined (mean difference = 0.7, P = 0.05, standard deviation [SD] = 0.39). Posttest scores were also significantly higher for each individual year (P = 0.01), with effect sizes ranging from 0.26 to 0.5. Table 5 and figure 3 show mean pretest and posttest scores for all four years.

FIG. 3. Mean pretest and posttest scores for student participants, 2012–2016 (n = 469).

TABLE 5. Means, SDs and effect size (Cohen's d) for 4-H Water Wizards student data 2012–2013, 2013–2014, 2014–2015 and 2015–2016

There were some variations in pretest and posttest scores by age. While fourth and fifth graders showed significant increase in knowledge (fourth-grade mean difference = 1.0; P = 0.05 and d = 0.49; and fifth-grade mean difference = 1.1; P = 0.01; d = 0.60), sixth graders did not show significant increase in knowledge overall.

Greater awareness of water issues and conservation. Participants who completed the program were more likely to identify two or more water issues in their communities in the posttest questionnaire than they did in the pretest. For instance, while 15% of participants named two water issues in the pretest (e.g., pollution or wasting water), this number increased to 24% in the posttest (a significant difference at P = 0.05). Also, the percentage of participants who cited no issues dropped from 33% in the pretest to 18% in the posttest (P = 0.05). Issues the students cited included “drought in Sacramento,” “people are wasting water,” “junk in creeks and lakes,” “fish are dying” and “running water for pools or leaving water on overnight.”

A majority of participants (79%) said they were using less water since participating in the 4-H Water Wizards program. When asked what they were doing to conserve, children most often gave simple, concrete examples like “[taking] shorter showers,” “turning water off when brushing my teeth” and “using less water by turning off the faucet when I am washing my hands.”

Project enjoyment. Overall, a large majority (85%) of the students agreed that they enjoyed the program. Comments from the afterschool staff confirmed this. Staff often commented that children loved the handson aspect of the program, that is, that they took advantage of the tools and license to explore activities.

Children were engaged in every activity, also were very excited to do hands-on activities.

Kids love hands-on science and learning.

Promising practices for science education in afterschool programs

Certainly, water is a critical resource that children in California, and throughout the world, need to understand. Afterschool programs can provide a gateway for exploring water science, especially in low-income communities. The 4-H Water Wizards program was designed to increase afterschool staff competency to teach water science to children, and raise children's knowledge and awareness of water. We conducted this four-year evaluation to determine if the program was effective in meeting its goals. In the discussion that follows, we explore the findings from our study and share thoughts about teaching science in the afterschool setting.

Understanding our findings in the context of afterschool programs

The nature of education in the afterschool setting is decidedly different than it is during in-school hours (Afterschool Alliance 2014). Attendance is not required, children may leave the program early and — having completed a day in the classroom before arriving — a child's attention is not optimal. The environment may be loosely structured, sometimes chaotic. For these reasons, it's a difficult setting for program fidelity, and scholastic outcomes are not always achieved (Deke et al. 2012; James-Burdumy et al. 2007). This may help to explain the average program effect size in our study.

Despite the obstacles, the 4-H Water Wizards project delivered reliable, if small, results. We found that there were consistent knowledge gains about water (with average effect size) overall for the participants of the program in every session. Fourth and fifth grade students showed statically significant gains in learning. The smaller sixth grade sample size, along with higher pretest scores, likely contributed to the non-significant results for this group. The consistency in results is especially significant when taken in the context of afterschool programs, where maintaining program fidelity is often a challenge. How did 4-H Water Wizards achieve consistent, positive program results in both knowledge and enjoyment despite these challenges in the afterschool context?

Expanded Learning (afterschool) program staff learn to measure water salinity using a hydrometer. Photo: Marianne Bird.

Integrating training and support

From our experience, when developing science programming for afterschool, creating curriculum and “handing it off” to program staff for implementation serves neither students nor staff. Curricula are most effective not as a stand-alone product but when designed with support for the practitioners (Ripberger and Blalock 2011). In 4-H Water Wizards, this included ongoing training, materials and coaching for afterschool staff.

Training sessions happened not once but three times during the 12-week program, reinforcing and building upon the material the staff was learning. Staff contributed questions and insight during these gatherings. The trainings also allowed participants to experience and deliver the content they would be presenting to students, thus building understanding and confidence in a subject many knew little about. During the training, staff shifted in their thinking about what was important in teaching science to children — from fun to exploration. This shift was likely related to the training, which emphasized science as exploration and investigation. Throughout the three trainings, staff saw demonstrations of inquiry-based teaching and, perhaps more importantly, had opportunities to practice and be coached. It appears that staff had lasting change in this area.

Also helpful was that the project included teaching materials. Program staff left each training session with all the materials needed to deliver content for the following unit. This eliminated the need for last-minute supply purchases or the inability to locate needed items, which might have led to skipped or compromised sessions. In addition, 4-H staff visited afterschool sites one or more times and checked in with afterschool staff to see where they were in the delivery of the project.

When reflecting on their own development throughout the project, afterschool staff articulated the value of student-constructed knowledge based in learn-by-doing experiences, as opposed to teacher-imposed knowledge based on simply telling information to children. Encouraging questions and investigation are foundational points in current methods of teaching science. This framing of how science is taught aligns with the adoption by most states, including California, of the Next Generation Science Standards (NRC 2012), which emphasize reasoning skills and concept development. The 4-H Water Wizards curriculum was written with a focus on inquiry, providing tools for facilitators in guiding learners in the process of discovery.

In 4-H Water Wizards, there is a bold focus on using ideas and energy from the students themselves to address water issues in the students' communities, and staff are trained to help this happen. The service-learning component — where children plan a community water-related project — is designed to help students apply their learning and see themselves as agents for positive change (Billig 2004). Our findings indicate that, prior to implementing 4-H Water Wizards, staff saw themselves at ease in empowering youth to be part of a project design but, when implementing the project, they realized challenges in doing so. This may not be surprising; working in partnership with young people is not commonplace in traditional educational settings. Future program design might include more training for staff to successfully integrate youth voices and partnership.

Conclusion

Afterschool programs provide excellent opportunities for nonformal, outside-the-classroom science education (Falk and Dierking 2010). Specific program practices may enhance the effectiveness of teaching science in afterschool, including training, support and program design. Our results suggest that a program design that includes ongoing engagement and collaboration with afterschool staff enhances successful learning outcomes in the afterschool context. Programs that utilize inquiry-based learning especially benefit from professional development that provides instructors with opportunities to experience what they will later replicate with students. This allows practitioners to construct their own understanding of content and process. In this regard, the tailoring of 4-H Water Wizards for afterschool settings is successful.

We do not know how the longer-term project goal — that of fostering concern and stewardship for the environment — plays out as young learners grow. It's a hopeful thought that an 11-session curriculum culminating in a single service-learning project would make a life-long impact on a child's behavior concerning water practices and greater care for the environment. Further study might explore the role of service-learning in cementing learning, changing behavior, encouraging environmental stewardship and developing engaged citizens.

Further research might also investigate if and how afterschool staff transfer inquiry-centered teaching when teaching beyond 4-H Water Wizards. Do afterschool staff transfer the science principles they learn in 4-H Water Wizards to other science activities in afterschool? Do youth retain information they've learned, and do they maintain water conservation practices over time? An exploration of such questions would further inform strategies to create optimal learning environments in afterschool settings.