"Attend our university, where we're not sure what--or even if--our
students are learning. We have dedicated faculty members and great
facilities, so we think we're doing a good job, but the truth is that we
have no evidence to support that."
Promotional copy like that
might draw some double takes, but not many applicants. Yet, allowing for
some hyperbole, this is the situation at many well-regarded
institutions of higher education.
As many experienced board
members know, given the national debate in recent years over
institutional accountability, learning--the heart of the educational
enterprise--is often treated as a by-product of other more measurable
processes. If the instructors are well-qualified, we assume that
students are learning. Or we measure the quality of an institution by
inputs, such as the admissions-test scores of entering students. But the
measure of educational quality should be determined by outputs or
outcomes--by what students have learned as a result of their educational
experience.
The fact is, no matter how excellent the curricula,
no matter how stellar the faculty, no matter how talented the student
body, without assessing student learning outcomes, there's no reliable
way to measure and demonstrate educational quality.
That's a
shame, especially for all of those colleges and universities with
excellent curricula, stellar faculty, and talented students. More
unsettling still is that without a way to demonstrate that learning
actually occurs in college classrooms, the entire postsecondary
enterprise rests on a shaky foundation.
Numerous assessment
measures have been developed in recent years, and several
higher-education associations have developed plans to help their member
institutions demonstrate their accountability. Yet evidence of student
outcomes is still less commonly gathered and publicized than "input
data" such as endowments and students' scores on admissions tests. More
governing boards and top administrators will be facing the need to take
assessment seriously in the future, however, given that accrediting
bodies increasingly are seeking evidence of student-learning outcomes.
But as many institutions have found out, obtaining sound evidence of
student learning--documenting what students know and can do--is not easy
and requires an institution-wide commitment.
What About Grades?
At this point, some people will respond, "We have a way to assess
student learning: It's called grades." And it's true that grades make
it possible to draw some inferences about relative student performance
in a specific class. But does Beverly's B- in Introduction to Psychology
mean that she learned more than Curt, who got a C+? What if Beverly and
Curt took the course from different professors? Or at different
institutions?
Student outcomes are integral to institutional
quality, and that quality is fundamental to fulfilling the missions of
the institutions that governing boards oversee. Without a reliable
measure of what students have learned, however, it is impossible to
determine if the institution is doing a good job. It is also impossible
to improve--or at least to determine whether improvements that have been
made are making a difference. And without standardized measures, it is
impossible for stakeholders to make conclusions about the relative
strengths and weaknesses of different programs or.institutions.
In
other words, providing and improving quality demands measurement. This
is not a new idea in business, which is accustomed to using metrics to
target, measure, and improve quality. But in higher education, the
notion of using metrics to define targets for student learning and
determine whether they have been met is sometimes considered
antithetical to academic freedom and institutional autonomy.
In
part, this attitude derives from the belief that learning is too
complex, too ineffable to be measured by a single test. This is true,
which is why institutions should create a balanced assessment portfolio
that reflects the multiple goals for learning at the institution. But
too often the problem is that faculty members and administrators are
simply uncertain about how to describe their goals for student learning
and transform them into measurable outcomes.
Creating a 'Culture of Evidence'
Over the past two years, the Educational Testing Service has
developed a framework for helping colleges and universities define
aspirations for student learning and translate them into measurable
outcomes. Assisted by a panel of educators, researchers, and educational
policy advocates, we published a series of white papers under the
collective title A Culture of Evidence (www.ets.org/cultureofevidence).
In
the first paper, we looked at the consequences for prospective
students, employers, and institutions of not having a systematic
approach to collecting evidence about student learning. Without such
evidence, we noted, it is difficult to see where changes are needed, to
examine the effects of such changes, and to engage in a process of
continuous improvement.
In particular, we looked at the need for evidence about student learning in four main areas:
- Workplace readiness and general skills;
- Discipline-specific knowledge and skills;
- "Soft" skills, such as teamwork, communication, and creativity; and
- Students' engagement in their own learning.
In the second Culture of Evidence report,
we looked at what is currently available to assess these areas. We
focused on the 12 most widely used standardized assessments, such the
National Survey of Student Engagement (NSSE) and ETS's Major Field Tests
(MFTs), and provided information about the instruments that would be
useful to colleges beginning or already engaging in developing a culture
of evidence.
In the third paper, we proposed a method for
helping an institution create a culture of evidence on its campus. This
method is based on ETS's own approach to test development, which begins
by asking what claims we want to be able to make on the basis of a test.
Similarly, the first step in the process of building a culture of
evidence is to determine what the institution wishes to be able to say
about what its graduates know and can do. This step is critically
important and must precede any attempt to find or build the "right
test."
Rather than impose an answer, we encourage institutional
stakeholders to examine their institution's mission and work together to
define the claims that they want to make about their graduates. For
example, do they want to be able to say that their students are prepared
to succeed in the global economy? If so, what does that mean? That they
can communicate effectively to a variety of audiences? That they have
an understanding and appreciation of diverse cultures? If so, what would
be the evidence for those claims? Such evidence could come in many
forms, from standardized assessments to presentations and portfolios.
Governing boards can be invaluable in.this process by asking key questions of.administrators and faculty members. For example:
- How will the administration and faculty members use the information gained from assessments to actually improve teaching and learning?
- Are the major employers of our graduates satisfied with new hires' knowledge and skills? If not, in what areas are they lacking?
- How do graduates of our institution compare with those of peer institutions?
Once
the claims that the institution currently wants to make about its
graduates have been defined and agreed upon, the next steps are to
conduct an "assessment audit" to determine what types of assessments the
college is currently conducting and whether they, in fact, support the
claims that the institution has identified as important. If additional
evidence is needed--if there are gaps between what the institution would
like to be able to claim about its students' learning and the claims
the available data can support--then the institution needs to consider
other assessments that might be useful.
Once assessments have
been administered and results are available, the crucial next step is
for the institution to examine the results in light of the aspirations
articulated in the first step. To what degree have the aspirations for
student learning been realized? Rigorous honesty is important for the
process to be useful. At the same time, it is essential to create an
atmosphere in which the results can be examined without fear of blame or
retribution. In such an environment, it is possible to ask what changes
need to be made to address learning shortfalls and to ensure that
current successes continue. The results of the data analysis need to be
communicated to all stakeholders, including board members.
As
this effort continues over time, the institution creates a culture of
evidence. It is a not linear process, but rather cyclical and iterative.
Aspirations and Autonomy
The "culture of evidence" process places a premium on the
autonomy of the institution and faculty members, the institution's
mission, and the populations and stakeholders the institution serves.
The process also encourages colleges and universities to reflect on what
aspirations they have for their students and then to generate evidence
that indicates how well they are meeting those goals. At the same time,
using reliable and valid standardized assessments as one component of
the assessment portfolio makes it possible to benchmark the
institution's performance against national and regional peers.
To
round out the picture of institutional effectiveness, trustees might
periodically ask the administration for other types of information on
how graduates are faring, such as job-placement rates, pass rates on
certification examinations for teaching and other professions, and
scores on graduate- and professional-school admissions tests.
Whether
the United States remains economically competitive will depend in part
on the ability of our postsecondary institutions to prepare
high-performing graduates equipped for the global workplace. America's
colleges and universities are still the world standard in higher
education. The challenge they face now is to build on their strengths
while focusing more on educational outcomes and demonstrating to
students the value of a higher education--that is, on creating a culture
of evidence of learning.
Once an institution has established and sustained a culture of evidence, it will be able to claim truthfully and with.pride:
"Attend
our university, where we create graduates who are prepared to succeed
in the global economy. And we have the evidence to prove it."
