|
The Culture of Evidence Internal Assessment The Universitys policy on program
assessment (UPS 410.200, effective December 12, 1992) makes the faculty responsible for
evaluation of academic programs. The vitality of the institution is dependent on the
commitment of its faculty. One form of commitment is a willingness to evaluate candidly
the programs and activities the faculty directs. Program Performance Review is a central
component of the evaluation. It is based on a thorough self-study which involves the
participation of the faculty. . .
The policy statement lays out the procedure for PPRs, specifying the responsibility of
school deans, the option of an outside reviewer, the requirement for a seven-year plan,
and the disposition of the report when it is completed. For programs that are accredited
by external organizations, that accreditation report may be substituted for a PPR.
Annual reports are required from each academic unit as well. In the past, annual
reports were fairly comprehensive, requiring an abbreviated vitae from each faculty member
covering the years activities, summary statements about curriculum changes,
sponsored events, student organizations and other activities including faculty
publications, research, and grants. More recently, the Vice President for Academic Affairs
has asked school deans to focus annual reports on specific topics, such as programs for
cultural diversity and, in 1998, efforts in assessment.
To prepare evidence for the Phase I report, we read each of the PPRs prepared in the
last five years (or accreditation reports where they substituted for PPRs) and the annual
reviews from each school for 1998 when assessment was one of the "required"
topics. We were searching for evidence of learning in our three theme areas and
documentation of outcomes assessment for individual programs. What we found was a wide
variety in approach to PPRs and such dissimilarity among the documents as to make
comparison difficult. Some PPRs had been constructed prior to or right at the adoption of
the universitys new Mission statement so a focus on mission was not present
in all the PPRs we read. We created speadsheets to facilitate comparison, and included in
them the categories we thought constituted the best documentation of program quality. We
looked for internal and external indicators, that is, at internal self-assessment and
assessment by important constituencies, such as alumni, employers, national agencies
(where appropriate), faculty peers, and students. Our spreadsheets attempt to summarize
our findings in the areas of overall program assessment and student learning outcomes.
We were guided in our search for evidence by a number of references, some helpful, some
skeptical. Our search helped us find gaps in our evidence (to be addressed in Phase II)
and areas where data exist but documentation as evidence for our three themes is less
clear. Publications from WASC and the American Council on Education, as well as materials
from sister campuses were especially useful.
Alexander W. Astins Assessment for Excellence: The Philosophy and Practice of
Assessment and Evaluation in Higher Education, links assessment to measurement and
evaluation for the purpose of what he terms "talent development." He maintains
the traditional definition of excellence in higher educationwhat he calls the
"resources" and "reputational" conceptions of excellenceare
flawed because they do not "directly address the institutions basic purposes:
the education of students and the cultivation of knowledge." Resources he defines as
money, high-quality faculty and high-quality students. He defines reputational as the
folklore that sustains the "pecking order" of excellence where those
institutions like Harvard, Yale and Berkeley are at the top and the rest of higher
education somewhere below.
Astin compartmentalizes traditional assessment into the four areas of admissions,
guidance and placement, classroom learning, and credentialing or certification, and then
evaluates each area to uncover its contribution to "talent development" of
students (we would probably call this "learning"). Only guidance and placement
procedures meet his criteria for "talent development" because effective guidance
and placement allow students to be assigned to classes appropriate for their level and
interests. Admissions procedures, credentialing and (an added entry) faculty evaluations
are not designed to help students but rather to further the reputation and resources of
the institution. Admitting students with outstanding high school records and high test
scores, evaluating faculty (primarily) on the basis of research publications, and seeking
prestigious credentials all enhance reputation (and garner resources). Classroom teachers
will be stunned to learn that Astin regards tests that measure classroom learning as a
form of credentialing that does nothing to help students develop their talents. Because
examinations are used primarily by instructors to assign grades, and because critical
examinations are frequently given at the end of a learning period when appropriate
feedback to the student is impossible, Astin finds this most common form of assessment an
ineffective tool to promote student learning.
Astins observations interested us because most of the PPRs we read did focus
strongly on the programs reputation and resources. Typically, the PPRs stress
faculty achievements in terms of research publications, external grants, or community
service. Translating these into tools for developing students talents, or learning,
is emphasized less.
An overview of assessment at the University of California, Santa Cruz written by Randy
Nelson, helped define more narrowly what kinds of data are available for measurement and
evaluation.
| In the current jargon of education, assessment is a broad term related to the
evaluation of educational effectiveness. In higher education, assessment includes
activities such as studies of potential students and non-matriculants; placement and basic
skills testing; surveying the educational goals and needs of new and continuing students;
learning why students drop-out or transfer; evaluating the need for and effectiveness of
student services; teaching and curriculum evaluation; surveys of the community; and
surveys of the alumni. Although the primary focus of assessment has been the undergraduate
student, evaluation activities now examine graduate students, faculty, and staff. |
Nelson says that assessment may be used to improve a process (formative evaluation) or
describe final outcomes (summative evaluation). Like Astin, he finds many university
practices (including classroom grading) to be summative rather than formative, doing
little to improve or enhance the learning experience. Nelsons report also covers the
political environment in California and the nation with respect to legislative mandates to
incorporate assessment into accountability systems. He mentions specifically the report on
student outcomes assessment produced by the CSU in 1989.
Student Outcomes Assessment in the California State University is a report to the
Chancellor by a special advisory committee on a year-long project to solicit campus views
and establish recommendations for assessment in the CSU. The report recommended the
establishment of a system-wide assessment policy that should be "campus-based,
faculty-centered, and student responsive." The report reflects the concern of the
advisory committee that introducing new assessment techniques not add significantly to
faculty workload, that they be well funded, and that student outcomes assessment "is
just one of several institutional practices that must exist in order to achieve
educational excellence" (page 13). The 10 member group included faculty and
administrators from five CSU campuses and its suspicion about the uses of outcomes
assessment is evident throughout the report. Citing political sources (the National
Governors Association, our own State Legislature, and particularly then Assemblyman,
now Senator Tom Hayden), the report acknowledged that accountability was an issue that was
not going to go away.
| "Higher education is a black box. You go in, and come out the other side. You
dont know what happened in it." |
Tom Haydens analysis reflects the Legislatures frustration with both the
CSU and the UC protection of turf. In turn, the Chancellors office responds with
mandates to do something about assessment (and accountability). Faculty respond that
"we have always done assessment" and become suspicious about what the new
"educational jargon" really means. The most persuasive portion of the
"Final Report" presented to the CSUF Academic Senate by its Ad Hoc Committee on
Assessment stresses the financial costs, increased faculty workload and unrealistic
expectations that incorporating assessment will bring.
Evidence of Student Learning: Data From 1998 Annual Reports on
Assessment
Suspicions aside, each school was required to submit a report on assessment of student
learning outcomes as part of its annual report to the Vice President for Academic Affairs
this year. What did the deans say about their schools? We can summarize quickly: Programs
that rely on external accreditation use the accreditation process as a major assessment
tool. Schools that house those programs are those most able to define assessment in terms
of student outcomes. Specifically:
Arts: Since all four programs in Arts are nationally accredited, all have
defined assessment to meet national standards. In all four programs (art, music, theater
and dance), students must audition and perform in some kind of juried setting or otherwise
submit their work for public critique. In several programs, external
reviewersincluding the local pressprovide feedback to students (and faculty).
"Most" art students are required to develop a portfolio for faculty review.
Business Administration and Economics: Like Arts, BAE is nationally accredited.
National guidelines require the school to establish measurable assessment goals. As the
schools next accreditation does not take place until 2002, BAE intends to start
developing its goals at its upcoming academic year.
Communications: Several programs in the school are nationally accredited. The Dean
reported that assessment would be on the agenda for the school retreat in August, 1998.
Both departments in the school indicated that a goal is to do more with assessment. The
Dean listed a number of assessment tools currently in use, including student portfolios in
several courses (assessed by external professionals), films (also assessed for film
festivals), work on the Daily Titan, which provides public exposure and
opportunities to be critiqued, internships, awards, and student competitions. The
Department of Communication was re-accredited in 1998 for five years. During the process
it used focus groups with students and alumni to discuss student learning outcomes and
reports that it received positive feedback.
Engineering and Computer Science: Programs were reaccredited recently by their
respective national associations but no assessment report was submitted this year.
Human Development and Community Service: The Dean submitted a five page report
outlining assessment activities in each of the schools divisions. Programs in
education are accredited by state and national agencies. Individual programs use
combinations of portfolios, capstone courses, and surveys of alumni and employers. Two
programs, Counseling and Human Services, participated in the Student Learning Initiative
this year to develop student learning outcomes for portions of their programs. The nursing
program is preparing for its accreditation next year. Kinesiology and Health Promotion
reported that assessment for its students are measured by employment rates, scores on
national tests, admittance to teacher education credential programs, and high evaluations
by internship supervisors (community professionals in the field).
Humanities and Social Sciences: One program in the school (the Masters of Public
Administration) is nationally accredited, and was last reviewed in 1996. The Dean cited
alumni surveys in five departments and a SWOT analysis in Liberal Studies. He also
indicated that defining the "marks" of a Fullerton graduate had assisted in some
development of measurable student outcomes. However, the dean reported no specific school
or department efforts to assess student learning.
Natural Science and Mathematics: The Dean stated that "informal assessment
is built into several" of the schools programs. He cited public colloquia where
students present research results, manuscripts jointly authored by faculty and students
accepted for publication in peer reviewed journals, portfolio-like laboratory journals,
and some exit interviews.
In summary, in 1998, schools reported some standardization in the methods used to
measure student learning outcomes, including, in many places, portfolios (or
portfolio-like products), public performances or presentations, alumni and employer
surveys, evaluation by external reviews (either through the accreditation process, or more
individually, student internships), and focus groups. However, with the exception of HDCS,
no dean reported a systematic effort to identify program goals and objectives and to tie
learning outcomes to those programmatic concerns. However, several deans did report that
effort is on their agenda.
PPRs and evidence of learning in theme areas
We looked at PPRs to find evidence of student learning, faculty and staff learning
and measures of the environment for learning. Our spreadsheets summarize our findings, but
the evidence offered by individual programs is singled out to demonstrate the different
approaches we found.
Student learning:
It may be easier to say what we did not find. We rarely found a focus on assessment,
particularly "student outcomes assessment," even as recently as this year, with
some exceptions.
The usual tools to measure student learning outcomes include standardized testing,
portfolios that are faculty reviewed, comprehensive examinations, theses, capstone
courses, performancesagain reviewed by faculty and in some cases external
reviewersand of course classroom-based testing and grading. We found no use of
standardized testing as an exit requirement, except of course for the Writing Proficiency
Test required of all. We found limited use of capstone courses, theses and comprehensive
examinations except at the graduate level. Portfolios, critiques and performances are
common in the School of the Arts, but infrequent elsewhere. We found wide spread use of
internships and other field work where students are assessed by "real world"
practitioners, and wide spread evidence of student excellence through competition for
awards, entrance into graduate programs, scholarships awarded, and successful employment
after graduation. Representative examples, all drawn from PPRs submitted between 1993 and
1998, follow.
School of the Arts: Music
- Students receive individualized attention regarding applied music lessons, jury process
of assessment, advisement and their course of study
- Intensive observation and internship experience in both the Music Education and
Piano-Pedagogy programs
- Formal advisement required of all undergraduates every semester
School of Business Administration and Economics: Accounting
- Student group won a regional meet to qualify for participating in the national round of
the Arthur Andersen Tax Challenge received a Honorable Mention at the national meet
- Of the leading accounting firms in Orange County, three of the managing partners are
graduates from the CSUF Accounting Department
- The CFO at Transamerica is a CSUF accounting alumnus

Members of the marketing team that won the General Motors Marketing Internship
National Scholastic Achievement Award competition celebrated during the 1997 $3,000
scholarship award presentation to the School of business Administration and Economics.
This represented the third year a CSUF student team won the contest's top prize, which
recognizes superior achievement in forming a marketing agency to research, design,
present, execute and analyze a promotional event for a local General Motors dealership.
Taking part in the presentation were Dr. Katrin Harich, who taught the managing
advertising class; then Dean (now Vice President of Academic Affairs) Ephraim Smith, and
former Vice President of Academic Affairs Mary Kay Tetreault.
School of Communications: Communications
- 112 graduates involved in mentoring 165 seniors
- One internship for credit is required during studentssenior year
School of Human Development and Community Service: Human Services
- 10% of majors graduated with honors, high honors or highest honors during the review
period
- Internships are an integral part of the curriculum:
| Students must enroll in a three-semester sequence of field placements while
concurrently enrolled in a fieldwork seminar Contracts with learning objectives and
assessment measures for the internship are developed for each student
|
|
|
|
- 90% of majors indicate plans for graduate work
Kinesiology
- Graduates have gone on to become athletic directors, administrators of elder-exercise
programs, coaches, etc. at well-known institutions across the nation
- Several recent graduates have gone on to Ph.D. programs
Education Division
- Extensive commitment to multicultural education found by the National Council for
Accreditation of Teacher Education (NCATE)
- Advisement/monitoring processes in the initial credential program are fully integrated
into the program
Elementary and Bilingual Education
- Students are provided with a strong link between initial coursework and field work
Reading Program
- Leadership skills of students are evidenced by examples of successful grant writing and
consulting activities
Secondary Education
- Students experience every aspect of teaching before actually beginning their student
teaching experience
School of Engineering and Computer Science: Computer Science
- Mandatory yearly advisement for all students
School of Humanities and Social Sciences: Afro-Ethnic Studies
- A large number of majors are double majors
American Studies
- The American Papers, a journal produced through the joint efforts of its student
editorial board and the journal advisor, publishes high quality student work
- Three students awarded Graduate Equity Fellowships
- Two students awarded California Pre-Doctoral Fellowships
- During the review period, a student received the Giles T. Brown Thesis of the Year Award
- Over a dozen students have delivered papers at professional conferences
- Examples of awards received by students:
- The H&SS Life Time Achievement Award
- The MacNeel-Pierce Oral History Scholarship
- A University of Hawaii Historic Preservation Scholarship
- Department recruits graduate students from a national constituency
Anthropology
- 19 graduates entered doctoral programs during the seven year review period
- Five students were recipients of the Chancellors Pre-Doctoral Fellowship Award
- Students have organized symposia for the Southwestern Anthropological Association (SWAA)
meetings
- A student won the first prize for the "Best Student Paper" at the SWAA
meetings in 1994
- Student participation in national and regional meetings of the Anthropological
Association
- In 1990, one student received the $3,000 National First Prize Award from the Lambda
Alpha Anthropology National Honor Society
- Three students received the Jenkins Award of Excellence from the Lambda Alpha Society
Chicano Studies
- One-third of survey respondents went on to graduate school
- 30% of these respondents enrolled in a credential program
English and Comparative Literature
- Enrollment in graduate program up 60%
- Significant number of graduate students invited to present papers at regional, national
and international conferences
- The South Coast Poetry Journal in operation until 1995 rated by one reviewer as
"among the best of the university-based literary magazines"
- The Jacaranda Review is a journal produced by student editors in conjunction with
English 408 which is a course that provides practical pre-professional experience
- 21st in the nation in graduating Hispanic-Americans with B.A.s in English
- 49th in the nation in conferring M.A. degrees on Asian-Americans
Geography
- Graduate student ranks tripled over the previous five-year review period
- M.A.s awarded at 60% over the previous review period
- Five graduates delivered papers at national meetings
- Three students continued on to Ph.D. programs
- Four recent alumni teach geography at area community colleges
Latin American Studies
- Dual language proficiency requirement (Spanish/Portuguese), the most rigorous of any
university in California
Political Science and Public Administration
- Several graduates chosen as Presidential Management Interns
- Half of the surveyed alumni have completed or are enrolled in a post-baccalaureate
program. Of those students who have completed, enrolled or considering graduate work, 51%
have pursued the Masters degree, 28% the pursuit of a J.D., 5% doctoral work, and
13% a teaching credential
- In the public administration MPA program, students must demonstrate skills in at least
two major computer applications upon entering the program those lacking this
knowledge must develop a plan for completion
- An internship is required of those without administrative experience in a public sector
agency in the MPA program
Psychology
- 794 students participated in independent study and directed research projects between
1987 and 1992
- A large percentage of graduates from the masters programs are currently enrolled
in a Ph.D. program or have received a Ph.D.
- In the academic year 1992-93 a psychology student received the Presidents
Associates Award and another student garnered the H&SS Life Achievement Award
- Nearly all graduate students intend to continue in Ph.D. programs
School of Natural Science and Mathematics: Biology
- 275 graduate students involved in research during the five year review period including
25 papers with students as co-authors, 25 published abstracts, 79 papers presented
- An average of 32 undergraduates per year participated in faculty-guided research
- 33 extramural proposals funded which specifically incorporated student research
- 92% of biology students recommended for admission to health professional schools are
accepted
- 85 graduating students have been the recipients of 14 different awards during the review
period
- 3 students have won a total of five research competitions
- Biology student won the Giles T. Brown, CSUF Outstanding Thesis Award
- Graduate program increased by about 25% during the review period
Chemistry
- Approximately 45% of bachelors degree recipients enter graduate and professional
programs
- Mandatory advising program
- 24 undergraduates and 21 graduates co-authored publications with the faculty during the
five year review period
- As noted by the external reviewer, a key strength of the program is the involvement of
undergraduate majors in research
- 154 undergraduates participated with faculty in research projects during 1994-95
- 53 students as co-authors at professional meetings during 1992-93
- 79 students as co-authors at professional meetings during 1993-94
Mathematics
- Two teams of students have entered the Mathematical Competition in Modeling
- In 1992, a mathematics student listed as a "top participant" in the Putnam
Competition
- In 1992-93, a student was awarded 2nd place in the Statewide Undergraduate
Research Competition
- In 1993-94, students won 1st and 2nd prize at the Statewide
Undergraduate Research Competition
- Student presentations at regional meetings and in statewide research competitions
- Several Paul Douglas Scholarships ($5,000) obtained for prospective teachers from the
department
Faculty and staff learning:

Acting Dean of ECS Richard Rocke and Professor David Cheng (Electrical
Engineering) confer with students and representatives from Lockheed Martin about one of
the 10 grants awarded for the CSU Partnership Program
Evidence from the PPRs of faculty learning invariably took the form of a
compilation of the research publications, conference papers, and internal and external
funding of grants by the programs faculty members. Some programs reported that
faculty had upgraded their technology skills, utilizing courses offered on campus. Others
reported that faculty attended programs sponsored by the Institute for the Advancement of
Teaching and Learning. Many programs reported that their faculty engaged in outside
consulting, professionally related service in the community and service on campus in
academic governance or as mentors. Programs could cite such evidence as development and
revision of course materials and innovative pedogogical applications. But, mostly, it was
publications that mattered. Those programs that received outstanding teaching awards did
mention them, of course.
Learning by staff in academic departments was almost never mentioned, although many
programs did acknowledge that excellent staff support contributed to the success of their
programs.
The following examples are all drawn from PPRs submitted between 1993 and 1998.
School of the Arts: Music
 |
The faculty exemplify excellence across a broad spectrum of
numerous specialties through performances, external professional reviews and employment in
professional activities |
|
- Strong recruiting and prospective-student-relations initiatives
|