(Pix (c) Larry Catá Backer 2015)
Learning outcomes assessment has become a very popular concept among those who seek to assert a measure of control over the university and its "product" (understood as students well equipped for insertion into appropriate segments of labor markets usually as a function of the position of the university within the prestige hierarchies in education). (See here). Sectors with a particular interest in asserting such control include political elites (for any number of reasons, judgment of these "reasons" and "justifications" is beyond the scope of this post), governmental ministries and the functionaries set to that task (usually in the form of some sort of governmental accreditation or standardization sub ministry), organizations of professionals seeking to preserve the integrity of the professions over which they assert power to set qualifications therefor (particularly the medical, legal and other professions), and of course university administrators who might see in the business of assessment a means of broadening cultures of control of the learning factories that are being constructed from out of universities in the early 21st century.
And, of course, it may be useful to faculty as well--perhaps as a means of communal self discipline, of personal improvement, as a means of retaining coherence in courses of study. Yet, within the context in which it has arisen, faculty have been the most reluctant stakeholders to embrace this approach to the management of the business of education. Timothy Reese Cain, an associate professor at the University of Georgia’s Institute of Higher Education and a senior scholar at the National Institute for Learning Outcomes Assessment has recently produced an Occasional Paper worth considering. Timothy Reese Cain, Assessment and Academic Freedom: In Concert, Not Conflict (National Institute for Learning Outcomes Assessment Occasional Paper No. 22 Nov. 2014). He argues that it is possible to construct systems of learning outcomes assessment that complement rather than threaten the now century old cultures of shared governance, tenure and academic freedom that are the hallmark of elite American universities.
This post considers in a very preliminary way his recommendations and their relation to efforts to construct systems of what ought to be a typical example of its type among research I universities--the learning outcomes assessment structures at Penn State. I will approach the later as if I were an uninformed outsider considering only information publicly available. Even this preliminary assessment suggests that if assessment becomes, in part, a system of evaluation not of courses
or of programs but of faculty, then the fears about the use of
assessment to end run both shared governance and academic freedom may
well be realized.
Almost 20 years ago, Anne E. Bilder and Clifton F. Conrad (1996) argued for the “wise use” of assessment results in graduate and professional education to protect “such cherished institutional traditions as academic freedom” (p. 12). Indeed, wisdom is needed throughout the entire process of assessment for all levels of students—from the articulation of outcomes statements to the selection and application of assessment measures to the ever-difficult loop-closing activities for improving student learning. How, then, might institutions pursue wise use in assessment to protect academic freedom? This pursuit will look different in the varying contexts and cultures of different campuses, but the key principles are the same for all institutions:
Protect and enhance shared governance more broadly. Academic freedom and shared governance are “inextricably linked” (Gerber, 2001). Without the latter, the former is substantially threatened. Faculty must be free to weigh in on plans, critique measures, discuss uses, and otherwise voice their opinions in both formal and informal ways. A culture of shared governance and full faculty participation—not just in assessment but more broadly—can only contribute to authentically documenting student accomplishment in ways that do not threaten academic freedom.Educate all stakeholders about academic freedom. Academic freedom is an essential value in, and a defining characteristic of, American higher education. Yet notions of it are often fuzzy and there is concern about complacency among the faculty and disdain by administrators. Making academic freedom an explicit and robust part of campus conversations about assessment but, again, also much more broadly—including the socialization of faculty and administrators in graduate education—is important if academic freedom is to be understood and enacted. Educating stakeholders about this should include not only the rights of faculty but also the negotiated limits of academic freedom—because academic freedom has never meant “anything goes.” Moreover, in shared curricular decisions, the rights of the faculty as a group can in some circumstances take precedence over the rights of individual faculty (AAUP, 2013).Put assessment fully under faculty purview. Assessment experts, whether from the faculty or not, are important. They can bring knowledge, help educate faculty, coordinate institution-wide efforts, and help provide the context and framing that make data useful. At the same time, to protect the faculty’s academic freedom, the outcomes defined, plans designed, and practices enacted must be under faculty control. As the former leaders of the three largest faculty unions have all argued, learning outcomes assessment itself is not a threat to academic freedom but in practice, when removed from faculty control, it surely can be (Gold et al., 2011).Have flexible plans that embrace disciplinary differences. Much of the recent concern over academic freedom in assessment has centered in the humanities, in which faculty fear highly standardized approaches that would miss the various types of learning they are trying to foster. Their concerns highlight the need to rely on disciplinary knowledge and faculty expertise to design and implement the most appropriate learning outcomes and assessment measures in different fields. Broader institutional goals, while certainly needed, should not be narrowly defined. Likewise, the assessment measures chosen need to attend to and emanate from the disciplines. As Laura Rosenthal (2010) wrote in discussing her work on assessment in the arts and humanities at the University of Maryland, College Park, “What most instructors I have talked to who are engaged in assessment projects will agree on is that there can be no one-size-fits-all model. Projects need to be specific to the institution, the department, and the discipline” (p. 155).Make use of what faculty are already doing. Early assessment efforts frequently were external to classes, partly to avoid intruding on classroom decisions that were the responsibility of individual faculty. Yet as the movement has matured, the importance of course-based evidence of learning has become more widely appreciated. Faculty are already constructing assignments that require students to think critically, to communicate effectively, and to demonstrate their learning. Student work for these assignments, when thoughtfully captured and considered, can form a basis for the larger assessment of student learning. Such an approach is not only efficient, it respects faculty and protects them from being required to do something additional or different when they are already providing evidence of learning (Hutchings, Jankowski, & Ewell, 2014).Frame assessment in terms of improvement—and mean it. Assessment has too often been driven by the need to meet accreditors’ standards rather than by an internal desire to improve. That has resulted in rushed efforts to create outcomes statements and gather evidence. Such an approach inhibits thoughtful discussions fully involving faculty and may lead to adopting other institutions’ models and methods without sufficiently considering their local applicability and appropriateness—creating conditions that minimize faculty participation and threaten their control over the curriculum and its enactment. Authentically framing assessment around improvement, as Jeremiah Ryan (1993) argued two decades ago, is not only important for making it useful but also for creating the context for inclusive conversations and faculty ownership.
Use the results wisely. The use of assessment results to improve student learning continues to be a key challenge. Far too often, outcomes are identified, measures and procedures are chosen and implemented, data is generated, reports are written—and little more comes of the work. Improvement of learning outcomes must be the primary purpose of assessment but, as Bilder and Conrad (1996) argued, that use must be wise and the inclination to force faculty to adopt new pedagogies that appear successful must be avoided. The results of assessment can help educate faculty and administrators, suggest new practices, and inform curricular revisions. They should be used to generate thoughtful discussion and encourage improvement but not to force the adoption of new styles or techniques and not to mandate changes in the classroom.
Timothy Reese Cain, Assessment and Academic Freedom: In Concert, Not Conflict (National Institute for Learning Outcomes Assessment Occasional Paper No. 22 Nov. 2014) pp. 12-14.
At Penn State, a senior administrator is quoted as suggesting that "Program and learning assessment resonate with what we as faculty do in our own scholarly research. We try creative new approaches, but need to conduct careful analyses and apply appropriate metrics to evaluate whether they are really working and achieving the outcomes we anticipated and want to see." (see here). But resonance is not engagement. It might even suggest its opposite, an assessment framework imposed from the top in which only operationalization is trickled down to the "worker" level. The Penn State Assessment website includes the following information:
Since 2005, Penn State has embarked on processes designed to assess and improve student learning within five spheres of assessment, including academic program assessment (aka learning outcomes assessment - LOA), course assessment, institutional assessment, general education assessment and co-curricular assessment.Penn State has developed quite useful tools in aid of developing assessment structures. See here and here. But these are the product of administrative design and targeted to suit the needs and expectations of outside governmental monitors and internal administrative objectives. How does this program architecture fit within the principles articulated by Professor Cain? Well, imperfectly it seems. Let's apply each in turn.
The purpose of this website is to inform administrators, faculty and staff of assessment initiatives as well as to support academic program assessment efforts. The site is under the purview of the Assessment Coordinating Committee (ACC), a committee of the Administrative Council on Undergraduate Education, and managed by the Schreyer Institute for Teaching Excellence.
Academic Program Assessment
Members of the ACC have developed expectations and due dates for the academic program assessment process at Penn State. ACC members partner with Schreyer Institute consultants to review assessment plans and provide feedback. Schreyer Institute consultants are available to consult with individuals or groups who are developing data collection plans or interpreting assessment data.
Protect and enhance shared governance more broadly. It is not clear that the Penn State learning outcomes assessment project has much to do with shared governance. Certainly there are no formal mechanisms through which the structures of learning outcomes assessment, lodged within the administrative apparatus of the university, has much connection to the institutional representation of the faculty. (See here). One can suppose that well chosen faculty may be selected to engage in some form or another of the work of the administrative structures deployed for the purpose of producing assessment mechanisms. But that sort of faculty engagement hardly counts for shared governance (see here and here). Indeed, the architecture of assessment at Penn State is notably top down in its construction, though the expectation is that the work of assessment is to be bottom up in the sense that faculty workers are responsible for the construction of their own assessment subject to their management from above. (see here).
Educate all stakeholders about academic freedom. There seems to be little by way of effort to make academic freedom an explicit and robust part of campus conversations about assessment, either as to its role as the responsibilities of faculty within its structures. There is at best only an implied recognition, without any sense of its context or principles, of the notion that in shared curricular decisions, the rights of the faculty as a group can in some circumstances take precedence over the rights of individual faculty (AAUP, 2013). That is found in the section of the Penn State Assessment FAQ ("A good way to start the program assessment process is to appoint an assessment coordinator or committee. The committee can then begin by identifying program goals."). That makes sense to the extent that assessment is focused on programs (over which there is or ought to be shared responsibility) as well as for individual courses (Penn State Assessment FAQ(Why do we have to do this?)). As administrative committees, there is neither incentive to protect or promote shared governance or academic freedom, especially when these might get in the way of administrative efficiencies. In hierarchical systems like those at Penn State, the critical site for education of academic freedom and shared governance, the critical site for developing administrative mechanisms for accountability, and transparency fall on the administrative structures which control the frameworks of assessment and their use.
Put assessment fully under faculty purview. As the analysis above suggests, responsibility for assessment is shared in the sense that the overall framework is managed by an administrative apparatus at the university level, but that operational control is devolved to units and data harvesting is the responsibility of individual faculty. It is hard to suggest that this system amounts to full faculty control of the assessment process. The reverse might be a more accurate assessment. There is a suggestion that the cause of this lies outside the university and falls on the diktat of the Middle States Commission on Higher Education, which recommended that Penn State “document the development of a comprehensive plan for the assessment of student learning that is linked to the overall assessment of institutional effectiveness and to the institute's strategic planning process.” (Penn State Assessment FAQ).
Have flexible plans that embrace disciplinary differences. It is in this respect that the Penn State Learning Outcomes Assessment project appears to be strongly focused on a disciplinary approach to assessment. The administrative mechanisms appear to encourage assessment based on program level assessment. That is to be lauded. (Penn State Assessment FAQ). Penn State distinguishes among (1) Academic program assessment , (2) Institutional assessment, (3) General Education assessment, (4) Cocurricular assessment , and (5) Course assessment. Exemplar assessment programs may be found here.
Make use of what faculty are already doing. Penn State's program does focus strongly on course assessment. This well serves faculty and provides a solid basis for what Professor Cain references as "the larger assessment of student learning." Yet it is not altogether clear that such course based assessment "respects faculty and protects them from being required to do something additional or different when they are already providing evidence of learning (Cain, supra, citing Hutchings, Jankowski, & Ewell, 2014)." The reason for that is inherent in the way in which assessment at Penn State is (not irrationally) hierarchically arranged. Within that pyramidal structure that is grounded on hierarchy, course assessment would occupy the lowest rung, below program, and institutional assessment. None of this is necessarily bad, but in the absence of substantial faculty participation it is possible, though of course inadvertent, for such assessment projects to attach academic freedom to course work, and then to subordinate academic freedom to the superior requirements of program and institutional assessment. That is precisely the danger that more robust faculty participation in assessment management would reduce.
Frame assessment in terms of improvement—and mean it.
To some extent Penn State's program exemplifies Professor Cain's critique of approaches "driven by the need to meet accreditors’
standards rather than by an internal desire to improve. That has
resulted in rushed efforts to create outcomes statements and gather
evidence." That is well evidenced by the focus on Middle States compliance at the heart of the Penn State program, at least as it is described in its own documents posted publicly. The focus on data generation and harvesting is also apparent though not surprising, as that seems inherent in a project of assessment. The Penn State FAQs state "Course and program assessment can provide data that will help course
instructors, programs, and departments make informed decisions in terms
of the program strengths and areas for improvement." Yet one might worry about the creation of what Professor Cain describes as "conditions that minimize faculty
participation and threaten their control over the curriculum and its
enactment." And indeed, one might worry as well of an assessment system that appears poised to serve as another means of evaluation rather than of assessment. The Penn State FAQs send quite mixed signals in that respect. On the one hand it provides that "In the context of higher education, the ultimate goal of assessment is to improve student learning." On the other it states that "the
data collected for assessment can also be used for other purposes such
as annual reviews." And in this ambiguity, traditionally deployed by university administrations nationally to make it possible to achieve functionally what ideology (including the ideology of academic freedom) constrains that the greatest danger lies. If assessment becomes, in part, a system of evaluation not of courses or of programs but of faculty, then the fears about the use of assessment to end run both shared governance and academic freedom may well be realized.
Use the results wisely. It follows then, that assessment used unwisely can be transformed from system to discipline programs and courses on an objectives based framework, to a system of disciplining and controlling faculty. That, of course, is inherent in a factory model of education; it is of little relevance to government ministries whose own governance logic makes them indifferent to the cultures of education they oversee. Professor Cain warns that institutions ought to resist the "inclination to force faculty to adopt new pedagogies that appear successful must be avoided" through assessment processes.It is not clear that this is adopted at Penn State. Rather it appears that mandatory data generation may be used by administrators in a variety of ways that may have the effect of pressuring change in the way faculty teach, and without the benefits either of shared governance constraints or with a sensitivity to the principles of academic freedom that might constrain such uses. What is provided is an ambiguous statement: "According to Walvoord (2004), assessment of student learning is conducted in order to determine what faculty as a whole can do to improve learning of students in their program. It should not be used as an evaluation of an individual faculty member. When there is evidence of inadequate student learning, faculty members and the department should collectively take appropriate action to address the issues and make improvement. In addition, end-of-course evaluations, aggregated across program faculty, could produce some useful information (Suskie, 2009)." (Penn State Assessment FAQ).
Penn State is not unique in its administrative heavy approach to assessment, and of the need to structure its operation through a model based on central control and decentralized data production and harvesting all of which is removed from academic freedom and shared governance by its fundamental characterization (developed by the administrators who run these ministries within the central administrative apparatus of the university) as an administrative, ministerial, technical or administrative in character. See, e.g., Ohio State. Indeed, the development and operationalization of learning outcomes assessment structures has become a multilateral effort among university administrations--none particularly mindful of the role of faculty except as data generators and objects of assessment and evaluation programs.
Indiana State University is one of 68 colleges and universities in nine states participating in a pilot initiative to assess student achievement of cross-cutting learning outcomes important to all disciplines. During the initial phase, student achievement in critical thinking, qualitative literacy, and written communication will be assessed using authentic student work submitted as a part of regular course assignments, such as projects, papers, and exams. The project is intended to produce meaningful data to assist the university to enhance student success. The Multi-State Collaborative to Advance Learning Outcomes Assessment (MSC) is sponsored by the State Higher Education Executive Officers (SHEEO) and the Association of American Colleges and Universities (AAC&U), working in partnership with the nine participating states and institutions. Additional information about the MSC is available here. (Indiana State University assesment).These are all based on a model of efficiency, harmonization and transportability--effectively moving toward a model of education in which teaching can be standardized to objectives and made portable, comparable and substitutable among institutions and education providers. Faculty are de-centered (see here) in this model in favor of the objectives and approaches that are now the markers of the commodity made available to students.
The Multi-State Collaborative to Advance Learning Outcomes Assessment (MSC) is an nitiative designed to provide meaningful evidence about how well students are achievingTo the extent that these assessment structures and objectives then have an underlying policy objective and effect, the role of faculty and the implications for academic freedom, ought not to be minimized or swept aside on the basis of false characterization of assessment as merely administrative, ministerial, technical or the like.
important learning outcomes. Sponsored by the State Higher Education Executive Officers (SHEEO) and the Association of American Colleges and Universities (AAC&U), the initiative foregrounds a distinctly different form of assessment than the traditional standardized test. Instead of producing reports about average scores on tests, the project is piloting the use of common rubrics applied by teams of faculty to students’ authentic college work—including such things as projects, papers, and research. The MSC is designed to produce valid data summarizing faculty judgments of students’ own work, and also seeks to aggregate results in a way that allows for benchmarking across institutions and states. The primary goal of the initiative is to provide data that will allow faculty and institution leaders to assess—and improve—the levels of student achievement on a set of cross-cutting outcomes important for all disciplines. (The Multi-State Collaborative to Advance Learning Outcomes Assessment)
No comments:
Post a Comment