Program Evaluation

for Five Academic Years (1992-1998)

Executive Summary

 

·      Career outcomes: A total of 85% of those surveyed one year post-program had obtained full-time employment as a CHW; received a promotion; or become a full-time student (94-97).

·      Student Proficiency: Only 16% of those taking our Performance Exam passed the pre-test; while 82% passed the post-test. The p value was <.0005 (97-98).

·      Our retention rate was 84% (97-98).

 

 

In order to guide our program development, we have carefully evaluated the Community Health Worker training program annually for five academic years (1992-98). We have a total N of 149. This paper describes our evaluation on three indicators over the five years: career outcomes, student proficiency and retention.

 

The Community Health Worker Certificate was designed to meet the needs of a non-traditional student population that is poor and working poor. Eighty-seven per cent were African-American, Latina/o, Asian Pacific Islander or Native American. Our typical student was a low income woman in her 30s; with a high school diploma and having both work and family responsibilities; the first in her family to attend college.

 

These students have many burdens and few resources to give them time to study. Most come from public schools that had minimal resources. To succeed in college, therefore, took great effort. Our students made these efforts and did succeed. One student worked full time during the day, attended our program in the evening and slept in his car before going to his second job on the graveyard shift; mothers found time to study late at night or very early in the morning.

 

Among our success stories are a student who entered our program with a job restocking grocery shelves and now supervises an AIDS outreach unit; another who was released from prison and now supervises the Tuberculosis outreach unit at San Francisco General Hospital; a third who entered working part-time as a CHW and is now enrolled in a master?s program in public health at San Francisco State University; a fourth who had been homeless and addicted, entered our program as a part-time peer educator and is now a full-time CHW. The large majority must confront complex social problems ranging from recent homelessness to recovery from drug addiction to domestic violence. In the context of these multiple risk factors, a combined response rate of 81% one year post-program (94-97) is itself a remarkable achievement.

 

Indicator 1: Career Outcomes. Figure 1 depicts three years of results of a post-program survey conducted one year after students completed the core course sequence (81%, combined N of 55 students participated in the survey). A research associate not known to the students completed a telephone survey to determine which students:

 

   1. obtained full-time employment as a CHW (for those who entered without such  jobs) or

   2. received a promotion (for those who entered as CHWs), or

   3. became a full-time student.

 

As indicated on Figure 1, students of all three classes received high rates of job placement and promotion as a result of the CHW Certificate Program:

 

    * 64% (n=35) of students entered already employed as CHWs seeking professional development: of these 77% (n=27) received promotions

    * 36% (n=20) of students entered without jobs as CHWs: of these, 85% (n=17) obtained full-time employment as a CHW (N=12) or became full-time students (n=5).

 

 

 

Over the three years, a total of 85% of those surveyed reported a positive career outcome as just defined. Respondents attributed these outcomes to their participation in and completion of the CHW Certificate.

 

Indicator 2: Student Proficiency. With our first pilot group we completed a survey of supervisors? (n=12) and students? (n=22) assessment of mastery on competencies using a Likert scale, showing a significant improvement with a p of < .0001.

 

As our program was refined, we developed a much more sophisticated instrument, a pre- and post- Performance Based Examination using a format recommended by the World Health Organization. This was administered and scored by employer-nominated examiners unknown to students and entirely independent of our staff. Two examiners were veteran Community Health Workers; two were CHW supervisors.

 

Students served as their own historical controls. In a simulation of an encounter with a ?standardized client? in a clinic, students performed certain competencies while being evaluated by an examiner against a very detailed proficiency checklist. The scoring system was rigorous and validated by employers as reflecting performance standards required for employment. The examination was given to incoming students before classes in August, and was repeated under identical conditions nine months later in May, after the students completed ten units of core courses. The final N was 38 paired pre- and post-tested students. Only 16% passed the pre-test, while 82% passed the post-test. The p value was < .0005. We present details of the results of the Performance Examination.

 

Only 16% passed the pretest; 82% passed the post-test. "Passing" is defined as getting an A, B, or C.

 

When graded against a very detailed proficiency indicator checklist,

 

    * A means that the student performance meets or exceeds employer standards

    * B means that student performance is competent and effective

    * C means that student performance needs improvement

    * D/F means that student performance fails to meet employer standards

 

Indicator 3: Retention rate. A literature search yielded some comparative data which suggested that our program is doing quite well in terms of student retention rate compared to other programs with a similar student population. For example, the Community College of Allegheny County ran a Health Careers Program for low income individuals. That program experienced a drop-out rate of 63%. In contrast, our program had a drop-out rate of 24% in 94-95, which we have been able to whittle down to 16% in 97-98.

 

One of our central pedagogical issues has been the development of methods to encourage retention of a high-risk student population. The two most important methods we developed include a prerequisite course to improve selection/admissions and the use of learning teams in the classroom. Over the course of 1995-98 we perfected these methods and were able to successfully double the size of the class while keeping our retention rate at 84% (in 97-98 we admitted 45 and 38 completed the core sequence).

 

Table 1 provides an overall summary of outcome evaluation data for the CHW Certificate from 92-98.

 

fn: evaloverall93-98/5.13.1999/pr1

 

 

 

 

Table 1. Overall Summary of Outcome Evaluation

Data for CHW Certificate

 

Academic Year 1992-93
 1993-94 1994-95  1995-96  1996-97

Cumulative

 

1994-97

 1997-98
 Major Accomplishments Planning and Statewide Occupational Survey

Pilot Year:

 

10 Units

Expanded to 17 units

 Certificate Approved by Curriculum Committee Regional Occupational Survey   Completed Performance Exam
Outcome Measured  

Learning/

 

Student Proficiency

 Positive Career Outcomes Positive Career Outcomes Positive Career Outcomes Positive Career Outcomes Student Proficiency
Instrument Used  

Supervisor and Student

 

Self-ranking of Mastery of Competencies


(paired t-test)
Survey One Year Post-Program Survey One Year Post-Program Survey One Year Post-Program

Cumulative

 

Post-program Surveys 1994-97

Pre- and Post-Performance Exam

 

(paired t-test)
N
  

36

 

(24 students plus

 

12 supervisors)

 20
 26
 22
 68

45

 

students took the pre-test

Response Rate

 

percent and (#) of students

  

94% (34)

 

(22 students plus 12 supervisors)

 90% (18) 65% (17) 91% (20) 81% (55)

84% (38)

 

of students enrolled took the post-test
Results
(p < .0001)

72% (13)

 

Positive Career Outcome1

88% (15)

 

Positive

 

Career Outcome1

80% (16)

 

Positive Career Outcome1

80% (44)

 

Positive Career Outcome1

(p < .0005)

 

16% passed the pre-test

 

82% passed the post-test

Shaded area indicates survey on Career Outcome for academic years 1994-1997.

 

1Positive Career Outcome indicates that the students either obtained a CHW position, received a promotion, or continued full-time education

 

BACK