4-H is a youth organization in which young people are given opportunities to build confidence, learn responsibility and develop life skills. Youth make friends and share interests, ranging from building robots to raising rabbits, from designing Web pages to landscape design; and they undertake volunteer projects in their communities.

The 4-H programs in many states use record books as a tool to teach youth about record keeping, improve their projects and reflect on their achievements. The record book format varies by state, and in most cases it is a personal description and reflection of their own experiences rather than a public record.

Along with personal data such as age, grade level and years in the 4-H program, the California record book includes a personal development report (PDR) with a quantitative page documenting participation in a variety of 4-H projects and activities, such as the number of meetings and events attended, projects completed, presentations given and awards received. The record books include project pages and a personal narrative written by the member describing his or her 4-H participation. This open-ended narrative is intended to describe the totality of one's history and experiences in the program in a sequential fashion.

Participants in the Shasta County 4-H program exhibit their project lambs at the Shasta District Fair.

These record books represent a large commitment of time by youth and adult volunteers. As a collection of personal information describing program experiences, the record books also provide a potential data source for program evaluation that has been heretofore untapped in California and elsewhere. Since the record books contain personal reflections, most are not shared beyond the club level, which may be the reason that they have rarely been used in evaluation.

Little has been published regarding the use of record books in 4-H program evaluation. Diem and Devitt (2003) examined 89 record books of New Jersey 4-H members and reported on what youth recorded learning (such as goalsetting, subject matter knowledge, organization and public speaking). Previous 4-H evaluations have primarily used survey data for program evaluation, rather than record books (Astroth and Haynes 2002; Howard et al. 2001; Lerner et al. 2008; Seevers and Dormody 1995).

The goal of this project was to use available record book data to evaluate how well the 4-H programs in Shasta and Trinity counties were achieving the statewide 4-H goals and mission, “to engage youth in reaching their fullest potential while advancing the field of youth development” ( www.ca4h.org ). This mission is lofty but challenging as the basis for evaluation at the county level. It is difficult to determine whether youth have reached their “fullest” potential, and advancing the field of youth development is an important mission but unrelated to members' personal reflections. To facilitate the evaluation, a more concrete set of goals was needed.

TABLE 1. Personal development report (PDR) categories coded to Targeting Life Skills model items, 2005–2006 (n = 341)

For many years, the 4-H program has focused on the development of citizenship, leadership and life skills. The specific life skills cultivated were identified in the Targeting Life Skills model developed by Patricia Hendricks of Iowa State University (Hendricks 1996). The model, also known as the Iowa wheel, includes 35 skills related to “Heart” (relating to and caring about others), “Head” (managing and thinking), “Hands” (giving and working) and “Health” (living and being) (fig. 1). Life skills included in this model represent a diverse range of characteristics, from personal qualities such as empathy, self-motivation and resiliency, to specific skills such as planning and organizing, problem solving and keeping records.

These 35 life skills were used as the basis for our evaluation of quantitative components in record book data from Shasta and Trinity counties describing the types of activities in which young people participated. Qualitative narrative data describing personal program experiences was coded and analyzed according to a statement from the 4-H Web site summarizing the 4-H experience: “4-H enables youth to have fun, meet new people, learn new life skills, build self-confidence, learn responsibility and set and achieve goals!”

Questions we sought to answer with this project included: (1) are record books usable as a data source for program evaluation? and (2) how well is the 4-H program helping young people to develop citizenship, leadership and life skills? To answer the first question, we had to examine the record books available to see whether they contained the kind of information that could be used to provide data on the success of the program. Once the first question was answered affirmatively, we could analyze the data to learn about young people's experiences in 4-H.

Fig. 1. Targeting Life Skills model. Source: Hendricks 1996.

Coding record book data

Record books were submitted to the 4-H youth development program representatives in Shasta and Trinity counties on a volunteer basis, with the knowledge that they would be used for this project. We used data from the 2005–2006 4-H year (4-H runs from July 1 to June 30, with most youth enrolling at the beginning of the school year). From 919 members who participated, 341 record books were collected, 330 of which included narratives. The sample represented 37.1% of all youth in the 4-H programs in Shasta and Trinity counties, and the narrative data represented 35.9% of all participants. We used two components of the record books for evaluation, the personal development report (PDR) and the personal narrative, “My 4-H Story.”

Personal development report.

The PDR includes tables that are filled in to quantify participation in 26 types of activities or honors, for the current year and previous years. For this analysis, we used the data simply to indicate whether the youth had participated in the activity or not, rather than to determine how many times they had participated (i.e., as a binary rather than a continuous model).

We compared the activities listed in the PDR to the life skills described in the Targeting Life Skills model and identified which life skills each PDR item would help develop (table 1). The individual PDR items were coded with more than one of the Targeting Life Skills model items in most cases. For example, having a radio or television appearance was coded as developing communication, leadership and marketable skills.

Not all of the Targeting Life Skills items relate to the categories defined in the PDR. Therefore, not all of the skills could be identified and coded; of the 35, 25 were coded to one PDR category or more. Conversely, 21 of the 26 PDR items received Targeting Life Skills model codes, while five items received no code (participation in activities other than 4-H, attended event, other group achievement, and attending local club and project meetings).

Ten of the Targeting Life Skills model items (accepting differences, conflict resolution, resiliency, wise use of resources, service learning, managing feelings, healthy lifestyle choices, stress management, disease prevention and personal safety) may be developed through a variety of 4-H activities, but were not coded because the PDR categories were not adequate to definitively say that the young person had developed these skills. For example, a young person participating in a rocketry project may develop personal safety skills. However, no PDR item is exclusive to such a project, so it was not possible to say using the PDR data how many young people developed personal safety.

Personal narrative.

To examine qualitative experiences in the program, the personal narratives were coded according to the Web site statement of 4-H goals. After the research group reviewed and agreed upon the codes, three researchers individually coded the youth narratives. To ensure consistency, the group discussed any items about which a team member was uncertain. Data from coding the narratives was entered into a spreadsheet.

For example, if a narrative reported that the youth had fun, had a good time or enjoyed the program, he or she was coded as “having fun.” “Meeting new people” included meeting buyers for their animals, meeting people at places where youth volunteer, and making new friends through program participation. “Learning new life skills” included learning any of the 35 life skills listed in the Targeting Life Skills model. Youth who reported increased self-confidence as a result of their participation in the program were coded as having developed “self-confidence.” “Responsibility” included learning responsibility or becoming more responsible through 4-H participation. “Set/achieve goals” included youth who reported that they had set one or more goals during that 4-H year or reported achieving one or more goals.

Statistical analysis.

The narratives were hand-entered into Microsoft Word and then coded using Excel. Data were analyzed using SAS Institute software for quantitative and NVivo software (QSR International) for qualitative data analysis.

Demographics.

Participants ranged from 9 to 18 years old, with a median age of 13 (table 2). Most record books came from youth who had participated in 4-H for fewer than 4 years, although 40% had been in the program 4 years or longer. The majority of participants were female (57%). Eighty percent lived in Shasta County, while the remainder lived in Trinity County.

TABLE 2. Demographics of 4-H study participants, Shasta and Trinity counties, 2005-2006

The personal development record (PDR) is filled out by 4-H members to quantify their participation in activities and honors. This data and the accompanying personal narratives were used to evaluate whether the 4-H programs in Shasta and Trinity counties were meeting their goals.

TABLE 3. Youth who developed various citizenship, leadership and life skills, according to personal development report (PDR) data (n = 341)

Can books be used for evaluation?

The study's first component examined whether record book data was adequate to evaluate the 4-H program. Examination of the categories reported in the PDR and information reported in the narratives led us to conclude that the record books have data that can evaluate how well the program is meeting its primary missions of engaging youth in citizenship, leadership and life-skills activities, including many of the life skills identified in the Targeting Life Skills model. The PDR data provides a structured look at participation in specific activities, while the narratives filled in additional qualitative information about personal perceptions of program experiences.

These findings are not without limitations, however. The PDR categories described specific activities rather than the skills gained from those activities. Since the categories did not match items in the Targeting Life Skills model, the coding provides incomplete coverage of possible life skills gained. It is possible that specific projects may develop skills, for example stress management or managing feelings, but it is not possible to ascertain from the PDR.

The qualitative data drawn from narratives is also subject to limitations. These narratives were open-ended and did not necessarily represent the totality of the young person's experiences in the program. Since youth were not asked specific questions about their development, or about any of the individual item codes, the narrative can provide only a baseline for members who mention particular characteristics, rather than an accurate quantification of rates for variables examined in the narratives.

Evaluating 4-H program goals

PDR analysis.

Youth in the 4-H club programs in Shasta and Trinity counties typically report a variety of activities, most commonly related to community service, individual projects and local fair participation (table 1). Completing at least one project was the most common activity noted on the PDR (87%). Additional commonly reported PDR items included “community pride service” (community-service projects) and “project showing contest” and “project exhibit” (showing projects at fairs, judging contests and demonstrations).

Analysis of the PDRs indicated that a majority of young people in the club program participated in leadership (63%), citizenship (84%) and life-skills activities (92%) (table 3). Most members (59%) reported PDR items indicating that they were engaged in all three types — citizenship, leadership and life-skills activities — while 28% reported two of the three and 4% reported just one.

Younger participants (below grade 6) are not expected to participate in leadership positions in their clubs such as holding an office, so it is not surprising that such engagement was reported less frequently than citizenship or life skills. The proportion of members who reported leadership activities was higher when the analyses were restricted to older members: among members in grades 7 to 12, 70% reported at least one such activity.

Based on matching PDR data to life skills delineated in the Targeting Life Skills model, more than 90% of members developed the life skills of sharing, communication, planning and organizing, goal-setting, self-motivation, record keeping and self-responsibility (table 3).

Narratives.

In the 330 qualitative narratives evaluated, most youth reported having fun (65%) and learning new life skills (68%). About 44% reported setting or achieving goals, and 39% said they learned responsibility. About one in five reported meeting new people and 16% reported that they gained self-confidence through participating in the program. These qualitative results represent only the characteristics or experiences that the members chose to describe, and as such represent a minimal baseline. For example, it is possible that more than 16% gained self-confidence, but only 16% mentioned it in their open-ended narratives.

Pros and cons of coding methods

Record books are completed by most youth in many county 4-H clubs, and they provide substantial detail about the 4-H participant's experiences, but until now the record books have remained an untapped data source. We found that this currently existing data source can be used for program evaluation at the county level. Other counties could use this project as a model for their own evaluations. Coding the record books was time-consuming, particularly for the longer record books typically completed by older members, but once entered into the spreadsheet and proofed, the PDR data was easily accessible and quantified.

Results showed that the 4-H programs in Shasta and Trinity counties were meeting the goals of cultivating citizenship, leadership and life skills among most youth. One significant limitation of this study is that the data available in this version of the PDR does not specifically describe the skills young people gain, so the results are limited by the accuracy of the assumptions in coding the PDR to the Targeting Life Skills model. Also, no data was available in the PDR on the amount of time youth spent on particular activities, so duration and intensity could not be quantified.

While this analysis attempted to identify the likely skills that youth were gaining from their participation, the life skills enumerated may not precisely match the skills youth gained. Our coding scheme may have overreported or underreported the resulting life skills, and the imprecise coding of life skills is a limitation of our study. The PDR items included some specifics allowing for the coding of 25 of the 35 life skills identified in the Targeting Life Skills model. The record book format has changed since this data was collected, improving the ability to evaluate members' life-skills development in the future. Rather than quantifying a particular array of activities, the new record book format allows youth to identify their development of leadership, citizenship, community-service and communication skills, as well as individual project skills gained.

The response rate of approximately 37%— in this case the proportion of record books that were collected at the county level— is an additional limitation. It is possible that the record books gathered did not constitute a representative sample of all youth in the program, which could skew the results. However, the record books collected represented members from 22 different clubs, in the full range of ages represented in the program as well as members who participated in a wide variety of activities, so it appears to be a broad sample. The 4-H club programs in Shasta and Trinity counties are not representative of all youth participating in 4-H around the state, so these results cannot be extended statewide. However, other counties may use this method to evaluate their own programs.