Wednesday, January 22, 2014

Managing Perception and Creating Reality Through Data Management--The Battle Over the "Best" Way to Understand MOOCs

I have recently posted about the habits of lazy and perhaps bad management as an impediment to the  development of MOOCs as an important element of university activity (Bad Management and the University Administrator: Giving Up on MOOCs?) and, as well, about the way in which assessment regimes are quickly replacing rule making as a means of governing institutions like universities (and its ill effects on shared governance) (Rulemaking Through the Back Door--Using Assessment Tools to Shape Education).


(Pix (c) Larry Catá Backer 2014)


These two trends have made an appearance recently in the conceptual battles over the way in which the academy "understands" MOOCs.  Now the academic machinery has appeared to have been put in overdrive as academics begin to deploy strategically chosen premises and the data harvested in the service thereof, to support any number of ways of "definitively" understanding MOOCs.  The stakes are high.  As consensus develops around the "better" way of "understanding" MOOCs, these will be used as the foundation of policy choices by university officials (and the governmental regulators who oversee them) to make policy about the scope and direction of MOOC use within the academy. People love data--they pay less attention to the premises in the service of which it is harvested and deployed to support some conclusion or other.  Both may be altered to strategic good effect.

This is not to mock those efforts or to suggest, necessarily, bad faith, on the part of researchers,  Quite the reverse. It is their good faith that lends these efforts their power, and the willingness to avoid critical engagement--with "data". However, it does suggest both the contingency of these investigations, and the way they are necessarily embedded in the policy  debates around which they are conceived. This post notes one of those efforts,  Ho, Andrew Dean and Reich, Justin and Nesterko, Sergiy O and Seaton, Daniel Thomas and Mullaney, Tommy and Waldo, Jim and Chuang, Isaac, HarvardX and MITx: The First Year of Open Online Courses, Fall 2012-Summer 2013 (January 21, 2014). HarvardX Working Paper Series No. 1. This study is particularly interesting because it explores the effects of changing a core premise of MOOC evaluation, and determines, that changing that premise may have a substantial effect on measuring "value". It starts with an excellent analysis: Jennifer Howard, Completion Rates Aren't the Best Way to Judge MOOCs, Researchers Say, Chronicle of Higher Education, Jan. 22, 2014.  We can only hope that these reports will contribute to a more nuanced and helpful discussion of consequences--especially about the role of MOOCs in the university and the relationship of faculty to these.



Jennifer Howard
Jauuary 22, 2014

When it comes to measuring the success of an education program, the bottom line is often the completion rate. How many students are finishing their studies and walking away with a credential?

But that is not the right way to judge massive open online courses, according to researchers at Harvard University and the Massachusetts Institute of Technology. “Course certification rates are misleading and counterproductive indicators of the impact and potential of open online courses,” write the researchers in the first of a series of working papers on MOOCs offered by the two universities. (The Harvard papers can be found here, the MIT papers here.)

. . . . .

The papers released on Tuesday draw on data from 17 MOOCs offered by Harvard and MIT in 2012 and 2013. A number of academics have begun studying aspects of the MOOC phenomenon, but few academic papers have been published so far.

The first of the working papers, which was written jointly by researchers at both universities, provides an overview of the data from those 17 MOOCs. Some findings:
841,687 people registered for the 17 MOOCs from Harvard and MIT.
--5 percent of all registrants earned a certificate of completion.
--35 percent never viewed any of the course materials.
--54 percent of those who “explored” at least half of the course content earned a certificate of completion.
--66 percent of all registrants already held a bachelor’s degree or higher.
--74 percent of those who earned a certificate of completion held a bachelor’s degree or higher.
--29 percent of all registrants were female.
--3 percent of all registrants were from underdeveloped countries.

Some of these findings reinforce what others have already observed about MOOCs: Few of those who sign up for a course end up completing it. Most MOOC students already hold traditional degrees. Students who sign up for MOOCs are overwhelmingly male.

But looking at percentages such as the ones listed above is a bad way to try to understand MOOCs, the researchers told The Chronicle in an interview.

Completion rates make sense as a metric for assessing conventional college courses, said Andrew Dean Ho, an associate professor in Harvard’s Graduate School of Education and director of the university’s MOOC research. In a conventional course, the goals are generally consistent and well understood: Students want to complete the course and, eventually, earn a credential. The instructors want the same thing.

A MOOC is more of a blank canvas, said Mr. Ho. Some students who register for MOOCs have no intention of completing, and some instructors do not emphasize completion as a priority. Success and failure take many forms.

. . . .

In future studies, the researchers hope to classify registrants according to their reasons for taking a MOOC, “so we can judge the impact of these courses in terms of what students expected to get out of them,” Mr. Ho said.

. . . .
 __________



Andrew Dean Ho


Harvard University; Harvard University - HarvardX

Justin Reich


Harvard University - HarvardX; Harvard University - Berkman Center for Internet & Society

Sergiy O Nesterko


Harvard University - HarvardX

Daniel Thomas Seaton


Massachusetts Institute of Technology (MIT) - Office of Digital Learning

Tommy Mullaney


Harvard University - HarvardX

Jim Waldo


Harvard University; Harvard University - HarvardX

Isaac Chuang


Massachusetts Institute of Technology (MIT) - Office of Digital Learning
January 21, 2014

HarvardX Working Paper Series No. 1
MITx Working Paper Series No. 1

Abstract:     
HarvardX and MITx are collaborative institutional efforts between Harvard University and MIT to enhance campus-based education, advance educational research, and increase access to online learning opportunities worldwide. Over the year from the fall of 2012 to the summer of 2013, HarvardX and MITx launched 17 courses on edX, a jointly founded platform for delivering massive open online courses (MOOCs). In that year, 43,196 registrants earned certificates of completion. Another 35,937 registrants explored half or more of course content without certification. An additional 469,702 registrants viewed less than half of the content. And 292,852 registrants never engaged with the online content. In total, there were 841,687 registrations from 597,692 unique users across the first year of HarvardX and MITx courses. This report is a joint effort by institutional units at Harvard and MIT to describe the registrant and course data provided by edX in the context of the diverse efforts and intentions of HarvardX and MITx instructor teams.

__________

 

MITx Working Papers

Background
On May 2, 2012, the presidents of MIT and Harvard University stood side by side to introduce edX, a jointly owned, not-for-profit venture to deliver open online learning opportunities to anyone around the world with an internet connection. The goals of the enterprise include increased access to educational opportunities worldwide, enhancement of on-campus residential education, and research about effective technology-mediated education. The respective university efforts to achieve these goals are known as HarvardX and MITx.
“The potential of new technologies is presenting all of us in higher education with a historic opportunity: the opportunity to better serve society by reinventing what we do and how we do it. It is an opportunity we must seize.” - MIT President Rafael Reif
The first three MITx courses launched in the Fall of 2012, seven more courses launched in the Spring of 2013, and one in the Summer of 2013.  As of this date, fifteen more courses recently completed or are about to commence, and dozens more modules and courses are in development. Now that data for the first 11 MITx courses have been delivered and analyzed, this is an opportune time to examine these first offerings, in order to inform ongoing course design and research.
MITx is pleased to make these initial reports available to the public (in tandem with the respective HarvardX courses). They address simple questions across multiple courses: Who registered?  What did they do?  Where are they from?
  • HarvardX and MITx: The First Year of Open Online Courses (MITx Working Paper #1)
    mitx-harvardx-course-icon-all
    The first three MITx open online courses launched on the edX platform in September, 2012; seven more courses launched in Spring 2013. This report and its companion course reports examine these initial six course offerings — alongside the initial 7 HarvardX courses — in order to inform ongoing course design and research. Now that data has been delivered and analyzed, it is an ideal time to examine these initial offerings in order to inform ongoing course design and research.
Results from this paper will be presented at the Harvard Graduate School of Education, at 6pm on January 21, 2013.

Working papers 2 through 12 are individual course reports.  We strongly encourage reading these reports (and the companion HarvardX course reports) as a package to understand the full story of the HarvardX and MITx initiative in its first year:

at

No comments:

Post a Comment