Earlier this week, ITHAKA released its findings from a randomized study at 6 public universities of Carnegie Mellon University’s Open Learning Initiative’s Statistics course, as delivered in a blended format. Their research found no credible differences in course completion or student learning outcomes between the students who took the face to face or blended course. A link to the full report is here. While the Gates Foundation didn’t sponsor or fund the research, the results of this study have important implications for our the technology enabled learning and innovations work that we fund broadly via our Next Generation Models team or the Next Generation Learning Challenges partnership. While the results of this study aren’t necessarily novel – they do emphasize once again that the development of intelligent learning technologies or digital cognitive tutors are at an early stage. While there are many caveats that even the researchers point out (i.e., this is only a study about one OLI course), this research is of tremendous importance to those of us who care about evidence based policy and change.
- The cost and level of implementation difficulty in executing this type of a randomized research experiment in higher education are high. While this experiment involved six campuses, because of complex institutional research requirements and the voluntary nature of participation in the study by students, ITHAKA was only able to put together a sample size of 605 students (total student enrollment in the course across all six campuses was over 3,000 students). This sample size is nevertheless statistically significant. In comparison, only 5 studies cited in the oft-cited DOE meta-analysis on online learning had sample sizes over 400 students.
- ITHAKA worked with six public universities to create “control” and “treatment” groups at each institution to compare and measure student learning outcomes between a intro statistics course taught in the traditional face to face method vs. a course taught with OLI (coupled with a once a week face to face meeting). The institutions that participated in the study were: CUNY Baruch College and CUNY City College, SUNY Albany, SUNY Institute of Technology, UM Baltimore County, and Towson University (also part of the University of Maryland system). While these institutions don’t represent an effective sample of all American higher ed, they do represent populations and demographics have both large commuting and residential populations, large numbers of minority students and students from low socioeconomic status families. No community colleges were able to participate in the study despite ITHAKA’s attempts to include them.
- Of great importance: the study found “no statistically significant differences in learning outcomes between students in the traditional – and hybrid format sections…we can be quite confident that the ‘average’ effects were in fact close to zero.” Even when a variety of other potentially significant factors amongst various subgroups were evaluated by the researchers (i.e. student data vis a vis race/ethnicity, gender, parental education level, college GPA, number of hours worked for pay), they still found that “on average, students learned just as much in the hybrid formats as they would have had they instead taken the course in the traditional format.” Even across the various institutions they didn’t see large differences in outcomes. Instead, the research did reveal that students who took the blended course spent an average of 25% less time completing the course (so similar learning outcomes as the face to face course in 25% less time).