Assessing the Efficacy of Third-Party MOOCs in Hybrid Instruction

Summary of a presentation by Rebecca Griffiths, ITHAKA

Why did we start this study? It’s very difficult to engage students ‘in the wild’. They come in with a great deal of different backgrounds. So we’ve chosen to study the efficacy of MOOCs inside an institutional environment.

Secondly, we want to study the efficacy of instruction itself. There’s a lot of discussion of ‘online’ vs ‘in-person’ but we are looking more at hybrids.

Third some of our recent work on barriers to adoption: administrators and faculty still feel there is not a sufficient supply of high-quality materials to choose from. The supply is growing, but we think if there were more materials there would be more likelihood to work with the new formats.

So, why MOOCs? There is not only the high quality content, there are the interactive formats, and the potential for feedback loops for continual improvements.

These MOOCs were not designed to be used in institutional settings. They were designed to teach a public audience, not as a service to other faculty and students. So for that reason ITHAKA and the University of Maryland have formed an alliance to explore these questions.

We are asking specifically:

-          Can MOOCs be used to improve student outcomes – competition rates

-          What models of adoption are out there, and what are the potential benefits and challenges

-          What implementation challenges arise?

-          What can we learn about cost savings?

-           

To answer these questions, we will be conducting 14-15 test cases incorporating MOOCs into hybrid sections at the University of Maryland. For these test cases we will create local instances of the MOOC, a MOOC just for that instructor that only the students can enroll in, and where the instructors can release the material at their own rates then. We’ll have two sets of sections, those where the students take hybrid MOOCs and those where students do traditional face-to-face.

(SD - So - the study is to 'remove all elements of the MOOC that make it a MOOC, and then compare with traditional learning')

For the case studies, these will be more anthropological studies – we will document the experience of the instructors and the students working with these materials. Finally, across all of these test cases we will be doing cost analyses.

The process:

-          Students will not be randomly assigned, they have a choice between hybrid and face-to-face

-          We will collect student data on recruitment with their permission

-          Baseline measure at the beginning of the course, pretests in some subjects

-          Collate outcome measures – completion rate, pass rate, scores on final exams, student satisfaction survey

Costs: it always costs more to do something the first time than in subsequent times. Instructors are learning to teach in a new way, they’re creating new materials. And the comparison is with instructors who have taught the same way for many years. So we want to think about how the third or fourth instances of the courses will be different – these are predictions but they’re probably pretty good.

(Some examples of test cases – Intro to Sociology from Princeton at Frostburg State University and Community College Baltimore County, for example, or Intro to Philosophy at the University of Edinburgh “that never really had video before, just reading assignments”).

What have we observed so far? Can’t say much yet, the tests are still progressing. General reactions from faculty wondering about how to mix and reorder, wondering about global community.

What might the study tell us? Studying one MOOC will not tell us whether another MOOC is any good. We can at best disprove a negative – to show that MOOCs can in fact be used in a hybrid format.

Question: it’s very clear this study is focused on campus-local hybrid courses. This isn’t the newest thing. Do you see the use of the MOOC to provide content as being something new?

Respone: That’s a great question? We had faculty saying “how is this any different from me going to YouTube and pulling off videos?” EdX has resulted in really great content that isn’t really available yet. The MOOCs are tightly bundled packages of platform, content and assessment, and faculty the first thing they want to do is pull that apart. But I do think there’s something different about these comprehensive courses. And as was said in the last session, they have started a conversation that wasn’t taking place before.

We will be talking to faculty in each course, asking if the assessments are getting what they want to get out.

Question: you said you will measure outcomes by exams, but also experiences. What will you be asking in that dimension?

Response; We’ll ask them whether they enjoyed the course, how much they feel they learned compared to the face-to-0face section, I think we will expand on those and ask about interaction, what sort of activity they did that required synthesis, etc.

(May Lit @ Singapore Management Univ: If an instructor has already built good online materials and learning activities in an LMS, how easy is it to convert the stuff in the LMS to a MOOC course? Anyone has experience in this?

Stephen Downes: @ May Lit - both Blackboard & Desire2Learn have already run MOOC pilots (I was involved in the D2L one) - your best bet is to contact the LMS people and ask them about converting the course to a MOOC within the LMS (especially if you're not really doing a MOOC but just one of the hybrid deliveries being described in these talks)).