Friday, August 18, 2017

Efficacy Studies and the Education Sector

 In "A Primer on Effectiveness and Efficacy Trials" (Clinical and Translational Gastroenterology (2014) 5), Amit G Singal MD, MS, Peter D R Higgins MD, PhD and Akbar K Waljee MD, MS, explain:
Intervention studies can be placed on a continuum, with a progression from efficacy trials to effectiveness trials. Efficacy can be defined as the performance of an intervention under ideal and controlled circumstances, whereas effectiveness refers to its performance under ‘real-world’ conditions.1 However, the distinction between the two types of trial is a continuum rather than a dichotomy, as it is likely impossible to perform a pure efficacy study or pure effectiveness study. (source, see also here)
This method  appears to be on the horizon of those who influence the cultures of the education industry (here, here, here, and here). 

Goldie Blumenstyk has recently reported for the Chronicle of Higher Education about the release of papers from a first of its kind symposium on efficacy research hosted by the University of Virginia:

1. Colleges spend upward of $5 billion a year on educational-technology products, but often they lack data that could better inform the decisions they make on what to buy. Over the past year, several dozen academics, business executives, and policy wonks researched why “efficacy research” isn’t more of a factor in these decisions. Some of those findings were presented at a symposium in May, and now the full reports are available.

2. Efficacy research isn’t just missing in ed tech. It’s also all-too-absent when it comes to the burgeoning world of “alternative” educational credentials, at least according to a new report by Ithaka S+R, a nonprofit consulting organization, for the American Academy of Arts & Sciences. Among other recommendations to policy makers, funders, and the higher-education community, the report recommends broadening quality-assurance processes so they can include educational programs not offered through traditional colleges as well as an investment in “a more comprehensive data system that captures longitudinal, student-record data on students’ experiences across the full array of postsecondary pathways, as well as information about providers and their programs and credentials.” In a world where some advocates are still pushing for more complete data on students in traditional higher-education settings, that could be a big ask. Or perhaps it will become one more argument in their favor. —

The links to the reports produced from the symposium and the press release follow. 

Symposium Reports

The Symposium Working Groups spent the past year collaborating to identify barriers, tackle complex challenges, and generate new insights to advance the field of edtech research and evaluation. Through surveys, interviews, and case studies, working groups explored issues such as how edtech decisions are made in K-12 and higher education and how companies and investors view and use research.

The Symposium Working Groups’ final reports summarize these findings and their implications for the role of efficacy research in the development, adoption, and implementation of edtech moving forward.
Working Group A
The Role of Research in K12 District Decision-Making
Working Group B
The Role of Research in Higher Ed Decision-Making
Working Group C
Research Spending & the Most Popular EdTech Products
Working Group D
Evidence and Quality of Efficacy in Research Approaches
Working Group E
Institutional Competence in Evaluating Efficacy Research
Working Group F
Investors & Entrepreneurs
Working Group G
The Goals and Roles of Federal Funding for EdTech Research
Working Group H
Education Philanthropies
Working Group I
EdTech User Voice
Working Group J
Crowdsourcing Efficacy Research and Product Reviews


First-of-Its-Kind Symposium Releases Working Papers to Advance Efficacy Research in EdTech

Year-long initiative brought together leading researchers, university and K-12 district leaders, faculty, entrepreneurs, and investors in education to explore the impact of edtech
WASHINGTON, DC—July 19, 2017—The University of Virginia’s Curry School of Education, Digital Promise, and the Jefferson Education Accelerator today released a series of papers developed by a cohort of ten working groups during a year-long collaboration focused on the role of efficacy research in education technology. The papers reflect a desire to improve the availability of efficacy research and the need for new policies and better data to inform the more than $13.2 billion of annual spending on education technology and ensure the technology effectively supports teaching and learning.

The first draft of these papers was presented during the EdTech Efficacy Research Academic Symposium, May 3-4 in Washington, D.C., which represented the culmination of 275 stakeholders who rarely collaborate in support of education technology: academics; researchers; entrepreneurs; district and university leaders; investors and philanthropists; and teachers and professors.

Leading up to the Symposium, each working group performed original research intended to identify the barriers to using research and other credible evidence in edtech decision-making.

“The Symposium uncovered some eye-opening structural problems in this space, as well as a series of paradoxical narratives that require unconventional alliances to address,” said Bart Epstein, CEO of the Jefferson Education Accelerator and Research Associate Professor at the University of Virginia. “Our research shows that educators and administrators nationwide want to understand why certain products thrive when implemented in one location, and fall short when implemented in another. And yet implementation research is neither funded at scale nor generally valued by the academic community because it is seen by government funders and academics as too commercial.”
“The net result,” said Aubrey Francisco, Chief Research Officer at Digital Promise, “is that educators and administrators have little actionable intelligence when it comes to making individual product choices. We heard loudly from Symposium participants that this needs to change.”

The Symposium papers also uncovered a serious collective action problem, according to Katrina Stevens, formerly Deputy Director of the Office of Educational Technology. “Nearly every institution wants to use research to make decisions,” said Stevens, “but no one educational institution or district has the incentive to invest in the sort of research necessary to test a product’s efficacy. New forms of collaboration are necessary to provide educators and administrators the information they need to make better decisions.”

At the Symposium, each working group issued working papers summarizing its findings and implications for the role of efficacy research in the development, adoption, and implementation of edtech moving forward. The reports explore a range of topics, including:
  • The Role of Research in K-12 District Decision-Making
  • The Role of Research in Higher Ed Decision-Making
  • Research Spending & the Most Popular EdTech Products
  • Evidence and Quality of Efficacy in Research Approaches
  • Institutional Competence in Evaluating Efficacy Research
  • Investors and Entrepreneurs
  • The Goals and Roles of Federal Funding for EdTech Research
  • Education Philanthropies
  • EdTech User Voice
  • Crowdsourcing Efficacy Research and Product Reviews.
“The event was the first-ever convening of stakeholders around the issues of efficacy research and the importance of elevating merit over marketing in edtech decision-making,” said Curry School Dean Robert C. Pianta. “With a better understanding of stakeholder perspectives and the barriers to collecting and using evidence of efficacy, we can now move to discussing the dozens of potential solutions raised for consideration by the Symposium participants.”
Each report is available for download on the Symposium website here:

The EdTech Efficacy Academic Research Symposium was made possible by Strada Education Network, with support from the Chan Zuckerberg Initiative, the Carnegie Corporation of New York, the Overdeck Family Foundation, the Bill & Melinda Gates Foundation, the Ewing Marion Kauffman Foundation, NewSchools Venture Fund, Gallup, Kaplan, Pearson, and Sterling Partners.
For more information about the Symposium or to be notified of future related events, please contact

About the Curry School of Education
The University of Virginia’s Curry School of Education, located in Charlottesville, Va., is ranked among the nation’s top 20 graduate schools of education. To its 2,300 undergraduate, graduate, and professional students the school offers nationally-ranked degree programs in education and health centered around human development.
Through 3 research centers, nearly 20 labs, and dozens of individual projects, faculty and students conduct rigorous, practical research that supports both the quality of teaching, learning and clinical practices, and the decision-making of district, state, and national leaders.

About Digital Promise
Digital Promise is a nonprofit organization that builds powerful networks and takes on grand challenges by working at the intersection of researchers, entrepreneurs, and educators. Our vision is that all people, at every stage of their lives, have access to learning experiences that help them acquire the knowledge and skills they need to thrive and continuously learn in an ever-changing world. For more information, visit the Digital Promise website and follow @digitalpromise for updates.

About Jefferson Education Accelerator
The Jefferson Education Accelerator (JEA) brings together the know-how, capital, and resources to address the unique needs of educators, entrepreneurs, and organizations. Combining the unabashedly innovative spirit of education technology with the academic rigor and excellence of one of the nation’s top-ranked education programs, JEA identifies the most promising growth-stage education companies, and helps them learn how to optimize their implementation models, set up new pilot programs at schools, colleges, and universities across the U.S., and raise capital that will put them on track to achieving scale and making an impact. For more information, visit  

Media Contact:
Sarah Herring,, 202-851-3606

No comments:

Post a Comment