The Culture of Evidence Internal Assessment

The University’s policy on program assessment (UPS 410.200, effective December 12, 1992) makes the faculty responsible for evaluation of academic programs. The vitality of the institution is dependent on the commitment of its faculty. One form of commitment is a willingness to evaluate candidly the programs and activities the faculty directs. Program Performance Review is a central component of the evaluation. It is based on a thorough self-study which involves the participation of the faculty. . .

The policy statement lays out the procedure for PPRs, specifying the responsibility of school deans, the option of an outside reviewer, the requirement for a seven-year plan, and the disposition of the report when it is completed. For programs that are accredited by external organizations, that accreditation report may be substituted for a PPR.

Annual reports are required from each academic unit as well. In the past, annual reports were fairly comprehensive, requiring an abbreviated vitae from each faculty member covering the year’s activities, summary statements about curriculum changes, sponsored events, student organizations and other activities including faculty publications, research, and grants. More recently, the Vice President for Academic Affairs has asked school deans to focus annual reports on specific topics, such as programs for cultural diversity and, in 1998, efforts in assessment.

To prepare evidence for the Phase I report, we read each of the PPRs prepared in the last five years (or accreditation reports where they substituted for PPRs) and the annual reviews from each school for 1998 when assessment was one of the "required" topics. We were searching for evidence of learning in our three theme areas and documentation of outcomes assessment for individual programs. What we found was a wide variety in approach to PPRs and such dissimilarity among the documents as to make comparison difficult. Some PPRs had been constructed prior to or right at the adoption of the university’s new Mission statement so a focus on mission was not present in all the PPRs we read. We created speadsheets to facilitate comparison, and included in them the categories we thought constituted the best documentation of program quality. We looked for internal and external indicators, that is, at internal self-assessment and assessment by important constituencies, such as alumni, employers, national agencies (where appropriate), faculty peers, and students. Our spreadsheets attempt to summarize our findings in the areas of overall program assessment and student learning outcomes.

We were guided in our search for evidence by a number of references, some helpful, some skeptical. Our search helped us find gaps in our evidence (to be addressed in Phase II) and areas where data exist but documentation as evidence for our three themes is less clear. Publications from WASC and the American Council on Education, as well as materials from sister campuses were especially useful.

Alexander W. Astin’s Assessment for Excellence: The Philosophy and Practice of Assessment and Evaluation in Higher Education, links assessment to measurement and evaluation for the purpose of what he terms "talent development." He maintains the traditional definition of excellence in higher education—what he calls the "resources" and "reputational" conceptions of excellence—are flawed because they do not "directly address the institution’s basic purposes: the education of students and the cultivation of knowledge." Resources he defines as money, high-quality faculty and high-quality students. He defines reputational as the folklore that sustains the "pecking order" of excellence where those institutions like Harvard, Yale and Berkeley are at the top and the rest of higher education somewhere below.

Astin compartmentalizes traditional assessment into the four areas of admissions, guidance and placement, classroom learning, and credentialing or certification, and then evaluates each area to uncover its contribution to "talent development" of students (we would probably call this "learning"). Only guidance and placement procedures meet his criteria for "talent development" because effective guidance and placement allow students to be assigned to classes appropriate for their level and interests. Admissions procedures, credentialing and (an added entry) faculty evaluations are not designed to help students but rather to further the reputation and resources of the institution. Admitting students with outstanding high school records and high test scores, evaluating faculty (primarily) on the basis of research publications, and seeking prestigious credentials all enhance reputation (and garner resources). Classroom teachers will be stunned to learn that Astin regards tests that measure classroom learning as a form of credentialing that does nothing to help students develop their talents. Because examinations are used primarily by instructors to assign grades, and because critical examinations are frequently given at the end of a learning period when appropriate feedback to the student is impossible, Astin finds this most common form of assessment an ineffective tool to promote student learning.

Astin’s observations interested us because most of the PPRs we read did focus strongly on the program’s reputation and resources. Typically, the PPRs stress faculty achievements in terms of research publications, external grants, or community service. Translating these into tools for developing students’ talents, or learning, is emphasized less.

An overview of assessment at the University of California, Santa Cruz written by Randy Nelson, helped define more narrowly what kinds of data are available for measurement and evaluation.

In the current jargon of education, assessment is a broad term related to the evaluation of educational effectiveness. In higher education, assessment includes activities such as studies of potential students and non-matriculants; placement and basic skills testing; surveying the educational goals and needs of new and continuing students; learning why students drop-out or transfer; evaluating the need for and effectiveness of student services; teaching and curriculum evaluation; surveys of the community; and surveys of the alumni. Although the primary focus of assessment has been the undergraduate student, evaluation activities now examine graduate students, faculty, and staff.

Nelson says that assessment may be used to improve a process (formative evaluation) or describe final outcomes (summative evaluation). Like Astin, he finds many university practices (including classroom grading) to be summative rather than formative, doing little to improve or enhance the learning experience. Nelson’s report also covers the political environment in California and the nation with respect to legislative mandates to incorporate assessment into accountability systems. He mentions specifically the report on student outcomes assessment produced by the CSU in 1989.

 

Student Outcomes Assessment in the California State University is a report to the Chancellor by a special advisory committee on a year-long project to solicit campus views and establish recommendations for assessment in the CSU. The report recommended the establishment of a system-wide assessment policy that should be "campus-based, faculty-centered, and student responsive." The report reflects the concern of the advisory committee that introducing new assessment techniques not add significantly to faculty workload, that they be well funded, and that student outcomes assessment "is just one of several institutional practices that must exist in order to achieve educational excellence" (page 13). The 10 member group included faculty and administrators from five CSU campuses and its suspicion about the uses of outcomes assessment is evident throughout the report. Citing political sources (the National Governor’s Association, our own State Legislature, and particularly then Assemblyman, now Senator Tom Hayden), the report acknowledged that accountability was an issue that was not going to go away.

"Higher education is a black box. You go in, and come out the other side. You don’t know what happened in it."

Tom Hayden’s analysis reflects the Legislature’s frustration with both the CSU and the UC protection of turf. In turn, the Chancellor’s office responds with mandates to do something about assessment (and accountability). Faculty respond that "we have always done assessment" and become suspicious about what the new "educational jargon" really means. The most persuasive portion of the "Final Report" presented to the CSUF Academic Senate by its Ad Hoc Committee on Assessment stresses the financial costs, increased faculty workload and unrealistic expectations that incorporating assessment will bring.

 

Evidence of Student Learning: Data From 1998 Annual Reports on Assessment

Suspicions aside, each school was required to submit a report on assessment of student learning outcomes as part of its annual report to the Vice President for Academic Affairs this year. What did the deans say about their schools? We can summarize quickly: Programs that rely on external accreditation use the accreditation process as a major assessment tool. Schools that house those programs are those most able to define assessment in terms of student outcomes. Specifically:

 

 

Arts: Since all four programs in Arts are nationally accredited, all have defined assessment to meet national standards. In all four programs (art, music, theater and dance), students must audition and perform in some kind of juried setting or otherwise submit their work for public critique. In several programs, external reviewers—including the local press—provide feedback to students (and faculty). "Most" art students are required to develop a portfolio for faculty review.

 

Business Administration and Economics: Like Arts, BAE is nationally accredited. National guidelines require the school to establish measurable assessment goals. As the school’s next accreditation does not take place until 2002, BAE intends to start developing its goals at its upcoming academic year.

 

Communications: Several programs in the school are nationally accredited. The Dean reported that assessment would be on the agenda for the school retreat in August, 1998. Both departments in the school indicated that a goal is to do more with assessment. The Dean listed a number of assessment tools currently in use, including student portfolios in several courses (assessed by external professionals), films (also assessed for film festivals), work on the Daily Titan, which provides public exposure and opportunities to be critiqued, internships, awards, and student competitions. The Department of Communication was re-accredited in 1998 for five years. During the process it used focus groups with students and alumni to discuss student learning outcomes and reports that it received positive feedback.

 

Engineering and Computer Science: Programs were reaccredited recently by their respective national associations but no assessment report was submitted this year.

 

Human Development and Community Service: The Dean submitted a five page report outlining assessment activities in each of the school’s divisions. Programs in education are accredited by state and national agencies. Individual programs use combinations of portfolios, capstone courses, and surveys of alumni and employers. Two programs, Counseling and Human Services, participated in the Student Learning Initiative this year to develop student learning outcomes for portions of their programs. The nursing program is preparing for its accreditation next year. Kinesiology and Health Promotion reported that assessment for its students are measured by employment rates, scores on national tests, admittance to teacher education credential programs, and high evaluations by internship supervisors (community professionals in the field).

 

Humanities and Social Sciences: One program in the school (the Masters of Public Administration) is nationally accredited, and was last reviewed in 1996. The Dean cited alumni surveys in five departments and a SWOT analysis in Liberal Studies. He also indicated that defining the "marks" of a Fullerton graduate had assisted in some development of measurable student outcomes. However, the dean reported no specific school or department efforts to assess student learning.

 

Natural Science and Mathematics: The Dean stated that "informal assessment is built into several" of the school’s programs. He cited public colloquia where students present research results, manuscripts jointly authored by faculty and students accepted for publication in peer reviewed journals, portfolio-like laboratory journals, and some exit interviews.

In summary, in 1998, schools reported some standardization in the methods used to measure student learning outcomes, including, in many places, portfolios (or portfolio-like products), public performances or presentations, alumni and employer surveys, evaluation by external reviews (either through the accreditation process, or more individually, student internships), and focus groups. However, with the exception of HDCS, no dean reported a systematic effort to identify program goals and objectives and to tie learning outcomes to those programmatic concerns. However, several deans did report that effort is on their agenda.

PPRs and evidence of learning in theme areas
We looked at PPRs to find evidence of student learning, faculty and staff learning and measures of the environment for learning. Our spreadsheets summarize our findings, but the evidence offered by individual programs is singled out to demonstrate the different approaches we found.

 

Student learning


It may be easier to say what we did not find. We rarely found a focus on assessment, particularly "student outcomes assessment," even as recently as this year, with some exceptions.

The usual tools to measure student learning outcomes include standardized testing, portfolios that are faculty reviewed, comprehensive examinations, theses, capstone courses, performances—again reviewed by faculty and in some cases external reviewers—and of course classroom-based testing and grading. We found no use of standardized testing as an exit requirement, except of course for the Writing Proficiency Test required of all. We found limited use of capstone courses, theses and comprehensive examinations except at the graduate level. Portfolios, critiques and performances are common in the School of the Arts, but infrequent elsewhere. We found wide spread use of internships and other field work where students are assessed by "real world" practitioners, and wide spread evidence of student excellence through competition for awards, entrance into graduate programs, scholarships awarded, and successful employment after graduation. Representative examples, all drawn from PPRs submitted between 1993 and 1998, follow.

 

School of the Arts: Music

  • Students receive individualized attention regarding applied music lessons, jury process of assessment, advisement and their course of study
  • Intensive observation and internship experience in both the Music Education and Piano-Pedagogy programs
  • Formal advisement required of all undergraduates every semester

 

School of Business Administration and Economics: Accounting

  • Student group won a regional meet to qualify for participating in the national round of the Arthur Andersen Tax Challenge – received a Honorable Mention at the national meet
  • Of the leading accounting firms in Orange County, three of the managing partners are graduates from the CSUF Accounting Department
  • The CFO at Transamerica is a CSUF accounting alumnus

 

 

Members of the marketing team that won the General Motors Marketing Internship National Scholastic Achievement Award competition celebrated during the 1997 $3,000 scholarship award presentation to the School of business Administration and Economics. This represented the third year a CSUF student team won the contest's top prize, which recognizes superior achievement in forming a marketing agency to research, design, present, execute and analyze a promotional event for a local General Motors dealership. Taking part in the presentation were Dr. Katrin Harich, who taught the managing advertising class; then Dean (now Vice President of Academic Affairs) Ephraim Smith, and former Vice President of Academic Affairs Mary Kay Tetreault.

 

School of Communications: Communications

  • 112 graduates involved in mentoring 165 seniors
  • One internship for credit is required during students’senior year

 

School of Human Development and Community Service: Human Services

  • 10% of majors graduated with honors, high honors or highest honors during the review period
  • Internships are an integral part of the curriculum:
    Students must enroll in a three-semester sequence of field placements while concurrently enrolled in a fieldwork seminar

    Contracts with learning objectives and assessment measures for the internship are developed for each student

  • 90% of majors indicate plans for graduate work

 

Kinesiology

  • Graduates have gone on to become athletic directors, administrators of elder-exercise programs, coaches, etc. at well-known institutions across the nation
  • Several recent graduates have gone on to Ph.D. programs

 

Education Division

  • Extensive commitment to multicultural education found by the National Council for Accreditation of Teacher Education (NCATE)
  • Advisement/monitoring processes in the initial credential program are fully integrated into the program

 

Elementary and Bilingual Education

  • Students are provided with a strong link between initial coursework and field work

 

Reading Program

  • Leadership skills of students are evidenced by examples of successful grant writing and consulting activities

 

Secondary Education

  • Students experience every aspect of teaching before actually beginning their student teaching experience

 

School of Engineering and Computer Science: Computer Science

  • Mandatory yearly advisement for all students

 

School of Humanities and Social Sciences: Afro-Ethnic Studies

  • A large number of majors are double majors

 

American Studies

  • The American Papers, a journal produced through the joint efforts of its student editorial board and the journal advisor, publishes high quality student work
  • Three students awarded Graduate Equity Fellowships
  • Two students awarded California Pre-Doctoral Fellowships
  • During the review period, a student received the Giles T. Brown Thesis of the Year Award
  • Over a dozen students have delivered papers at professional conferences
    • Examples of awards received by students:
      • The H&SS Life Time Achievement Award
      • The MacNeel-Pierce Oral History Scholarship
      • A University of Hawaii Historic Preservation Scholarship
      • Department recruits graduate students from a national constituency

 

Anthropology

  • 19 graduates entered doctoral programs during the seven year review period
  • Five students were recipients of the Chancellor’s Pre-Doctoral Fellowship Award
  • Students have organized symposia for the Southwestern Anthropological Association (SWAA) meetings
  • A student won the first prize for the "Best Student Paper" at the SWAA meetings in 1994
  • Student participation in national and regional meetings of the Anthropological Association
  • In 1990, one student received the $3,000 National First Prize Award from the Lambda Alpha Anthropology National Honor Society
  • Three students received the Jenkins Award of Excellence from the Lambda Alpha Society

 

Chicano Studies

  • One-third of survey respondents went on to graduate school
  • 30% of these respondents enrolled in a credential program

 

English and Comparative Literature

  • Enrollment in graduate program up 60%
  • Significant number of graduate students invited to present papers at regional, national and international conferences
  • The South Coast Poetry Journal in operation until 1995 rated by one reviewer as "among the best of the university-based literary magazines"
  • The Jacaranda Review is a journal produced by student editors in conjunction with English 408 which is a course that provides practical pre-professional experience
  • 21st in the nation in graduating Hispanic-Americans with B.A.s in English
  • 49th in the nation in conferring M.A. degrees on Asian-Americans

 

Geography

  • Graduate student ranks tripled over the previous five-year review period
  • M.A.s awarded at 60% over the previous review period
  • Five graduates delivered papers at national meetings
  • Three students continued on to Ph.D. programs
  • Four recent alumni teach geography at area community colleges

 

Latin American Studies

  • Dual language proficiency requirement (Spanish/Portuguese), the most rigorous of any university in California

 

Political Science and Public Administration

  • Several graduates chosen as Presidential Management Interns
  • Half of the surveyed alumni have completed or are enrolled in a post-baccalaureate program. Of those students who have completed, enrolled or considering graduate work, 51% have pursued the Master’s degree, 28% the pursuit of a J.D., 5% doctoral work, and 13% a teaching credential
  • In the public administration MPA program, students must demonstrate skills in at least two major computer applications upon entering the program – those lacking this knowledge must develop a plan for completion
  • An internship is required of those without administrative experience in a public sector agency in the MPA program

 

Psychology

  • 794 students participated in independent study and directed research projects between 1987 and 1992
  • A large percentage of graduates from the master’s programs are currently enrolled in a Ph.D. program or have received a Ph.D.
  • In the academic year 1992-93 a psychology student received the President’s Associates Award and another student garnered the H&SS Life Achievement Award
  • Nearly all graduate students intend to continue in Ph.D. programs

 

School of Natural Science and Mathematics: Biology

  • 275 graduate students involved in research during the five year review period including 25 papers with students as co-authors, 25 published abstracts, 79 papers presented
  • An average of 32 undergraduates per year participated in faculty-guided research
  • 33 extramural proposals funded which specifically incorporated student research
  • 92% of biology students recommended for admission to health professional schools are accepted
  • 85 graduating students have been the recipients of 14 different awards during the review period
  • 3 students have won a total of five research competitions
  • Biology student won the Giles T. Brown, CSUF Outstanding Thesis Award
  • Graduate program increased by about 25% during the review period

 

Chemistry

  • Approximately 45% of bachelor’s degree recipients enter graduate and professional programs
  • Mandatory advising program
  • 24 undergraduates and 21 graduates co-authored publications with the faculty during the five year review period
  • As noted by the external reviewer, a key strength of the program is the involvement of undergraduate majors in research
  • 154 undergraduates participated with faculty in research projects during 1994-95
  • 53 students as co-authors at professional meetings during 1992-93
  • 79 students as co-authors at professional meetings during 1993-94

 

Mathematics

  • Two teams of students have entered the Mathematical Competition in Modeling
  • In 1992, a mathematics student listed as a "top participant" in the Putnam Competition
  • In 1992-93, a student was awarded 2nd place in the Statewide Undergraduate Research Competition
  • In 1993-94, students won 1st and 2nd prize at the Statewide Undergraduate Research Competition
  • Student presentations at regional meetings and in statewide research competitions
  • Several Paul Douglas Scholarships ($5,000) obtained for prospective teachers from the department

 

Faculty and staff learning:

Acting Dean of ECS Richard Rocke and Professor David Cheng (Electrical Engineering) confer with students and representatives from Lockheed Martin about one of the 10 grants awarded for the CSU Partnership Program

Evidence from the PPRs of faculty learning invariably took the form of a compilation of the research publications, conference papers, and internal and external funding of grants by the program’s faculty members. Some programs reported that faculty had upgraded their technology skills, utilizing courses offered on campus. Others reported that faculty attended programs sponsored by the Institute for the Advancement of Teaching and Learning. Many programs reported that their faculty engaged in outside consulting, professionally related service in the community and service on campus in academic governance or as mentors. Programs could cite such evidence as development and revision of course materials and innovative pedogogical applications. But, mostly, it was publications that mattered. Those programs that received outstanding teaching awards did mention them, of course.

Learning by staff in academic departments was almost never mentioned, although many programs did acknowledge that excellent staff support contributed to the success of their programs.

The following examples are all drawn from PPRs submitted between 1993 and 1998.

 

School of the Arts: Music

    The faculty exemplify excellence across a broad spectrum of numerous specialties through performances, external professional reviews and employment in professional activities
  • Strong recruiting and prospective-student-relations initiatives
  • Faculty in all areas of music present public recitals on campus
  • Strong faculty presence in local public schools through clinic, adjudication, and guest-conductor assistance

 

 

School of Business Administration and Economics: Accounting (# of faculty = 19)*

 

Economics (# of faculty = 19)*

 

School of Communications: Speech Communications

 

School of Human Development and Community Service

 

Human Services (# of faculty = 5)*

 

Kinesiology (# of faculty = 16)*

 

Education Division

 

Elementary and Bilingual Education

 

Reading Program

 

Secondary Education

 

School of Engineering and Computer Science: Computer Science (# of faculty = 12)*

 

School of Humanities & Social Sciences: Afro-Ethnic Studies (# of faculty=4)*

 

American Studies (# of faculty = 9)*

 

Anthropology

 

Chicano Studies

 

Comparative Religion

 

English and Comparative Literature

 

Foreign Languages and Literature (# of faculty = 21)*

 

Geography (# of faculty = 10)*

 

Latin American Studies

 

Political Science (# of faculty = 20)*

 

Psychology

 

School of Natural Science and Mathematics: Biology (# of faculty = 25)*

 

Chemistry (# of faculty = 23)*

 

Mathematics

 

Environment for Learning:

The new Anthropology offices in McCarthy Hall, refurbished by a $1,000,000 grant from NSF

Evidence relating to the campus environment for learning is the most diverse of the findings we have to report. Programs that used technology all needed upgraded equipment (these were all written before the rollout, of course). That was about the only universally shared opinion. Those programs that encouraged community -based programs and strong student support did mention those features. We don’t know if those programs that omitted such mention did so because they provide a less hospitable environment. Many programs mentioned faculty participating in mentoring. A few programs talked about outreach to community colleges and high schools. Very few mentioned strong relationships with Student Affairs programs or other resources around the campus. Several reported efforts to improve student access by soliciting money for scholarships, or obtaining research grants to support student work. This latter was especially true in the sciences.

Accreditation reports indicated that faculty are generally pleased with their physical surroundings, but PPRs rarely mentioned office space. (Kinesiology is an exception: it complained about the poor quality of its housing and surroundings.) No one talked about other amenities—or the lack of them—around the campus. Only a few even mentioned library resources.

Some sample comments:

 

School of the Arts: Music

 

School of Business Administration and Economics

 

Accounting

 

Economics

 

School of Communications: Communications

 

Speech Communications

 

School of Human Development and Community Service: Education Programs

 

Elementary and Bilingual Education

 

Secondary Education

 

Reading Program

 

Human Services

 

Kinesiology

 

School of Engineering and Computer Science: Computer Science

 

School of Humanities and Social Sciences: Afro-Ethnic Studies

 

American Studies

 

Anthropology

 

Chicano Studies

 

Comparative Religion

 

 English and Comparative Literature

 

Geography

 

Political Science and Public Administration

 

Psychology

 

School of Natural Science and Mathematics: Chemistry

 

Mathematics

 

A note about some "exemplary practices":

As we noted earlier, most of the PPRs we examined were written before new guidelines were in place for the 1997-98 period. However, we read the PPRs of two departments, Human Services and Chemistry, that anticipated the direction of the new guidelines. We called them "exemplary practices" because they serve as models for what we hope the new reports will be.

 

Human Services – 1993

The PPR prepared by the Human Services Department set out in black and white what the department views as learning objectives and what measures will be used to assess achievement of those objectives. Nine broad learning objectives such as developing an understanding of the diversity within client populations and acquiring knowledge and practice skills for intervention are spelled out with specific strategies included on how to achieve these goals. Assessment measures include research and theory-based position papers, group projects which entail community/field activities, journal writing in which students gauge the impact of the course’s content on their attitudes and values, and student discussion of specific cases and field work experiences. It is worth noting that this PPR was written in 1993 before the University’s Mission and Goal statement was in effect.

 

Chemistry – 1995

The format of the Chemistry Department’s PPR is patterned after the University’s Mission and Goals statement. Detailed and specific narratives of how the Chemistry Department achieves each of these goals are provided. Intertwined with the narratives are the Department’s philosophies and definitions of these goals.

One means of assessing student learning is to follow the paths of students after graduation, and this department does a remarkable job of tracking graduates and alumni after leaving CSUF. As stated in the introduction, the data truly have been compiled in such a way as to be useful to the Department, School, and University in preparing proposals; involving and tracking alumni; summarizing grant, contract, and publication activity; and providing a perspective on past and future activities.