Should you Mooc and match?

Another professor’s learning materials? In my course? It’s more likely than you think.

The non-profit research organisation Ithaka S+R this month released its highly anticipated report on its work with the institutions in the University System of Maryland, which for the past 18 months have experimented with courseware from Carnegie Mellon University, Coursera and Pearson in face-to-face courses.

Backed by a $1.4 million (£0.8 million) grant from the Bill and Melinda Gates Foundation, the study aimed to produce some sorely needed research about massive open online courses and their usefulness to brick-and-mortar institutions.

Eighteen months later, the Mooc frenzy has calmed, and Ithaka’s findings are similarly muted.

“Our findings add empirical weight to an emerging consensus that technology can be used to enhance productivity in higher education by reducing costs without compromising student outcomes,” researchers Rebecca Griffiths, Matthew Chingos, Christine Mulhern and Richard Spies write.

In other words, hybrid courses that mix online content with face-to-face instruction can be just as good, though not necessarily better, than traditional courses.

But a second research question - whether faculty members can use course content created by their counterparts at other institutions, potentially saving both time and resources - produced less clear results.

Generally speaking, faculty members in the 17 courses studied enjoyed the outside content, most of which came from Coursera. Of the instructors using Mooc content, only three opted to use the video lectures alone in their courses, while 13 used different kinds of content, including quizzes. In future courses, 15 instructors said they would use Mooc content, but only a single instructor answered no.

Individual case studies, however, paint a different picture. There, faculty members’ optimism is intermixed with concerns that content is dictating the flow of the course - and not the other way around.

In a comparative politics class, for example, “the content was not adequately aligned with [the instructor’s] course”. The same happened in an introductory genetics course, where the instructor was “forced … to restructure his course” to cover topics addressed in the Mooc. Students in an information literacy course also took longer than expected to get through the Mooc content “because some of the content did not align with [the instructor’s] course as much as he would have liked.”

Some faculty members, like the comparative politics instructor, said they would be likely to create their own videos in the future to avoid those complications.

“One key takeaway is that careful planning for the implementation of new course formats and technologies is absolutely critical, and the results of any individual test are influenced by a large variety of factors that have nothing to do with the technology,” the report reads. “Course redesigns can take several iterations to fine tune as platforms get better and instructors grow more comfortable with the technology.”

Students were largely indifferent about the new course materials. In one course, “students did not completely understand the purpose of integrating the Mooc” and struggled to see its connection to what they were being taught in the classroom. The instructor in the comparative politics course had expected students would be more interested in watching videos than reading textbook chapters, but by the end of the semester, only 10 of his 20 students had even opened the videos.

Other instructors noted that the students seemed less likely to retain knowledge from the video lectures, since they watched the clips on their own time. As the instructors spent less time with the students in the hybrid sections - 72 minutes, on average, compared with 126 minutes in face-to-face courses - there were fewer opportunities for them to stress key concepts.

Many of those hours saved were erased by the time it took to prepare the courses. On average, faculty members spent 175 hours getting their courses ready for the start of term, but the researchers indicate that a wider selection of course materials, experience and stronger institutional support could drive that number down.

“While many innovative faculty members - including those who participated in this study - have demonstrated great creativity and willingness to try new approaches with technology, these individual efforts can only be sustained and achieve large scale impact within a coherent strategic framework,” the report reads. “Ultimately, faculty must be willing to try using materials created elsewhere and be open to the potential benefits these materials can offer their students.”

Overall, students in sections that did not use any of the online content rated their courses higher than those in the hybrid sections. Those students also claimed they learned more.

The self-evaluations did not translate to actual course outcomes, however.

Although the difference was not statistically significant, students in the hybrid courses were more likely to pass the class, 87 per cent compared with 83 per cent, and they also slightly outperformed students in face-to-face courses, earning a final grade of 72 (compared with a 70). Perhaps more significantly, the researchers “found no evidence supporting the worry that disadvantaged or academically underprepared students were harmed by taking hybrid courses”.