Presented at the Eastern Educational Research Association Conference
February 25, 1999, Hilton Head, South Carolina

Please address correspondence to:
Teresa L. Flateby, Director
Evaluation and Testing
University of South Florida
4202 E. Fowler Avenue, SVC 2054
Tampa, FL 33620

 
Three years ago, this evaluator was invited to be an internal/external evaluator for a new Learning Community (LC) program at the University of South Florida, a metropolitan university of 34,000. While internal to the university, the evaluator was external to the college in which this program was delivered. The approach used to evaluate the LC program has evolved from a primarily external outcomes orientation, to the recently introduced participatory and collaborative process orientation for evaluating six LC’s.

Background and Initial Approach

Learning Communities were introduced at USF to provide a supportive environment for groups of 50 students enrolled in liberal arts courses team-taught by faculty from various disciplines. Writing-across-the-curriculum (WAC), in which writing is taught within the context of a particular discipline, is an important feature of the two-year program. Because this program was in its infancy, was evolving and expanding, it was reasoned that the evaluation should adapt to program changes. Therefore, a responsive evaluation approach (Stake, 1976) was introduced.

Initially, the program’s administrators who contacted the evaluator seemed mostly interested in describing the LC process and assessing the outcomes of the first LC. Based upon the stated goals of the learning community, both from written materials and communication with faculty and administrators, student satisfaction, classroom activities and writing were studied by this evaluator and a doctoral student. Multiple methods consisting of observations, questionnaires, writing samples and interviews were chosen to address the areas of investigation and to provide the potential for triangulating the results (Appendix A).

Program documents and information offered by the administrators and faculty were used to focus the observations (Patton, 1990). The doctoral student observed classroom teaching styles and environmental characteristics during the beginning, middle and end of each term in each LC. A two-part questionnaire was developed using printed program materials and information provided by the administrators and faculty. The preliminary questionnaire was distributed to the five LC faculty, administrators and students for their comments and suggestions. The questionnaire was revised accordingly. The resulting questionnaire surveyed LC students’ satisfaction with multiple aspects of the LC and compared LC and non-LC students’ satisfaction with their cumulative experiences at this institution. In order to evaluate the effectiveness of the writing-across-the-curriculum program for teaching writing skills, a timed writing sample was collected from students completing Freshman English I in traditional classes and in the LC. Writing prompts administered to LC and a sample of Freshman English students, were developed by the evaluator and the LC coordinator who was also a faculty member from the English Department. The essays produced from this exercise were scored holistically. In addition to the other types of data collected, a random sample of LC students was interviewed to determine their satisfaction with the various aspects of the LC.
 
 

First Evolution

During and immediately following the first year, it was realized from a review of learning communities literature and experiences with the LC that higher order thinking skills should be addressed. Since the integration of content was and continues to be central to learning communities as compared to more traditional and often fragmented approaches to instruction, an expected outcome of this instructional focus was the enhancement of higher order thinking skills. Galbelnick, MacGregor, Mathews and Smith (1990) stated that the integration of courses in learning communities allows students to "have opportunities for deeper understanding and integration of the material they are learning" (p.19). To reinforce this objective, Resnick and Resnick (1992) asserted that if thinking skills are encouraged, if students are expected to solve problems, make informed judgments or use any higher order thinking skills, these processes must be assessed. In response to this additional information and a greater understanding of the program, the evaluation was refocused to ascertain if faculty from LC 1, in its second year and LC’s 2 and 3 in their first year, encouraged these higher order thinking skills in the classroom and if these skills were evidenced in students’ writing. The questionnaire was revised slightly to reflect the shift in evaluation emphasis. Also, focus group interviews replaced individual interviews to mirror the community nature of the program and to elicit input from a broader sampling of students. A second doctoral student was added to assist with the evaluation process.

The evaluation was modified again during the third year to reflect the continued emphasis on higher order thinking skills. Focus group questions were developed to query students’ perceptions about the cognitive levels demanded in their LC. Also, observations collected for the five operational LC’s during that year focused upon the types of questions faculty asked students and students’ responses to the questions. An alumni questionnaire was developed and administered to students who had completed the LC the previous year.

Continued Evolution

The evaluator again responded to the needs and constraints of the LC program. First, the evaluation was becoming more and more complex and time consuming, as a result of the addition of new LC's each year. While the evaluator followed the "Guiding Principles for Evaluators" (1998), and communicated the results with program administrators, deliverers and recipients, these results have been used only to restructure the WAC component. Even though the evaluation became more process oriented, and the administrators were supportive of the evaluation efforts, few changes were implemented as a result of the evaluation. Most faculty did not appear to realize any implications for themselves or their specific LC. The administration generally maintained a hands-off approach to LC classrooms, and the faculty seemed to view the evaluation as an external product evaluation. In addition, the underuse of the results could have been affected by the timing of the distribution of the semi-annual and annual reports. Moreover, although the primary LC stateholders were informed and consulted during the design, data collection and reporting phases, their behavior indicated little to no identification with the evaluation or the recommendations offered.

In an attempt to encourage utilization of the results, the evaluation approach was modified before the 1998–1999 academic year to encourage participation and ownership of the evaluation and to reflect the process orientation and the collaborative nature of the program. According to Cousins and Earle (1992), five factors contribute to the success of participatory evaluation methodology. The organization must: 1) value evaluation, 2) provide the resources and time to complete the evaluation, 3) value organizational learning, 4) have primary program users motivated to contribute to the evaluation, and 5) have primary users who are willing and able to receive the necessary training to participate in the evaluation.

With the contributors to the successful participatory evaluation in mind, faculty, advisors, administrators and students from the six LC’s were asked to volunteer to participate on the evaluation team. The resulting team, representing all LC’s, disciplines and participants’ roles and functions, suggests that at least some members of the organization value evaluation and organizational learning and have made a time and learning commitment to the evaluation process. During the summer, the evaluation team: 1) refined and prioritized LC goals to be assessed, 2) identified methods and team members responsible for collecting the initial data, 3) provided a data collection schedule to all LC participants, and 4) designed a plan to distribute results in a more frequent and timely manner. Team members have begun to collect and report data. Training has been provided during regularly scheduled meetings prior to the collection of data from each method. Also, a web site has been developed to report on our activities, including our data collection schedule and program goals (See Appendix B) (http://www.usf.edu/~lc/assessment_team.htm).

Although this collaborative approach has been fruitful and enlightening, compromises have been necessary, requiring the evaluator to relinquish much control. Some participants volunteer for greater responsibilities than others, but all evaluation team members observe several LC’s, contribute to the modification of the questions to the questionnaires and interviews and participate in focus groups. The team is currently resolving how to assess writing this term and negotiating areas to evaluate next year. We have presented at several conferences and have benefited from this collaborative evaluation effort.
 

Participant Involvement

In this roundtable, evolution of the evaluation will be addressed and a table (Appendix A) highlighting changes and illustrating the participatory nature of the current evaluation approach will be distributed. The following questions will be posed to engage the roundtable participants: 1) have others altered their evaluation approach within a single project, and if so what were the attitudes of the participants and outcomes produced, 2) what problems should be anticipated as a result of changing the approach, and 3) are there suggestions to ensure success?

References

American Evaluation Association. (1998). Guiding Principles for Evaluators. [On line] Available: http://www.eval.org/EvaluationDocuments/aeaprin6.html.

 

Bloom, B.S.(1956). Taxonomy of Educational Objectives, New York, David McKay.

 

Cousins, J.B., Earl, L.M. (Winter, 1992). "The Case for Participatory Evaluation,"Educational Evaluation and Policy Analysis, Vol. 14, No. 4, 397-418.

 

Galbelnick, F., MacGregor, J., Mathews, R., Smith B.(1990). Learning Communities: Creating Connections Among Students, Faculty and Disciplines. New Directions for Teaching and Learning, N. 41, San Francisco: Jossey-Bass.

 

Olson, C. (1992). Thinking Writing: Fostering Critical Thinking Through Writing, Harper Collins, New York, 428-429.

 

Patton, M.Q. (1990). Qualitative Evaluation and Research Methods, Sage Publications, Newbury Park, California.

Resnick, L.B., and D.P. (1992). "Assessing the Thinking Curriculum: New Tools for Educational Reform in Changing Assessments." In B.K. Gifford, and M.C. O'Connor. Alternative Views of Aptitude, Achievement and Instruction. Boston: National Commission on Testing and Public Policy, 37-75.

Spear, K. (1983). "Building Cognitive Skills in Basic Writers." Teaching English in the Two-Year College, 9 (2). 91-98.

Stake, Robert E., (1976). Evaluating Educational Programmes: The Need and the Response, Organization for Economic Cooperation and Development. Paris, France.

Steele, J.M. (1997). "Identifying the Essential Skills in Critical Thinking at the Post-secondary Level to Guide Instruction and Assessment." [Draft]. Iowa City: American College Testing.