Evaluating a MOOC
I was asked (along with Dave Cormier and George Siemens):
"How might it be possible to show that cMOOCs are effective for learning, in the sense of providing evidence that institutions might accept so as to support opening up more courses to outside participants (a la ds106, Alec Couros' EC&I 831, etc.)? Or, more generally, providing evidence that participation in and facilitating cMOOCs is worthy of support by institutions... What I'm looking for are criteria one might use to say that a cMOOC is successful. What should participants be getting out of cMOOCs?"
I think the best way to understand success in a MOOC is by analogy with, say a book, or a game, or a trip to the city.
The person taking the MOOC is like a person reading a book, playing a game, or taking a trip to the city. It is impossible to talk about 'the objective' of such an activity - some people want to learn something (and others something else), others are doing it for leisure (and others as part of their job), others to make friends (and others to get away from their friends for a while), etc.
If we were a commercial enterprise we could focus on sales. Then we could focus an ad campaign on the actual reasons people take MOOCs - but we wouldn't need to worry about whether they were met, only about whether our advertising enticed people to pay the fee. But I think that's a pretty narrow criterion for success.
I would adopt George's suggestion, and look to the institutional goals for offering MOOCs. But again here we find a wide array of interests: some want to use MOOCs as advertising, to entice people to enrol in other courses; others want to experiment with new methods of delivering learning; others want to support products or services they sell; still others want to serve a social good and provide free learning from the community. Each objective will have its own metric for success.
My own response treats a MOOC for what it is: a network. I then ask whether it satisfied the properties of a successful network. I can do this from two perspectives: first, from a process perspective; and second, from an outcomes perspective.
The process perspective asks whether the MOOC satisfied the criteria for successful networks. Of these, the most important are contained in what I call the Semantic Condition, which ensures that the MOOC remains a living system. The semantic condition contains four parts: autonomy, diversity, openness, and interactivity. The MOOC is assessed against each of these and a degree of compliance may be found.
The outcomes perspective looks at the MOOC as a knowing system. By that what I mean is that the MOOC should exhibit network properties on a macro scale - in other words, that we should be able to say things about the MOOC without reference to particular individuals in the MOOC. This is to treat the MOOC as an entity which perceives, or which learns, as a whole. These things are emergent properties, for example, emergent knowledge or emergent learning. Did the MOOC as a whole produce some new insight, or recognize some new phenomenon in its area of study?
MOOC success, in other words, is not individual success. We each have our own motivations for participating in a MOOC, and our own rewards, which may be more or less satisfied. But MOOC success emerges as a consequence of individual experiences. It is not a combination or a sum of those experiences - taking a poll won't tell us about them - but rather a result of how those experiences combined or meshed together.
This may not reflect what institutional funders want to hear. But my thinking and hope is that over th long term MOOCs will be self-sustaining, able to draw participants who can see the value of a MOOC for what it is, without needing to support narrow and specific commercial or personal learning objectives.
"How might it be possible to show that cMOOCs are effective for learning, in the sense of providing evidence that institutions might accept so as to support opening up more courses to outside participants (a la ds106, Alec Couros' EC&I 831, etc.)? Or, more generally, providing evidence that participation in and facilitating cMOOCs is worthy of support by institutions... What I'm looking for are criteria one might use to say that a cMOOC is successful. What should participants be getting out of cMOOCs?"
I think the best way to understand success in a MOOC is by analogy with, say a book, or a game, or a trip to the city.
The person taking the MOOC is like a person reading a book, playing a game, or taking a trip to the city. It is impossible to talk about 'the objective' of such an activity - some people want to learn something (and others something else), others are doing it for leisure (and others as part of their job), others to make friends (and others to get away from their friends for a while), etc.
If we were a commercial enterprise we could focus on sales. Then we could focus an ad campaign on the actual reasons people take MOOCs - but we wouldn't need to worry about whether they were met, only about whether our advertising enticed people to pay the fee. But I think that's a pretty narrow criterion for success.
I would adopt George's suggestion, and look to the institutional goals for offering MOOCs. But again here we find a wide array of interests: some want to use MOOCs as advertising, to entice people to enrol in other courses; others want to experiment with new methods of delivering learning; others want to support products or services they sell; still others want to serve a social good and provide free learning from the community. Each objective will have its own metric for success.
My own response treats a MOOC for what it is: a network. I then ask whether it satisfied the properties of a successful network. I can do this from two perspectives: first, from a process perspective; and second, from an outcomes perspective.
The process perspective asks whether the MOOC satisfied the criteria for successful networks. Of these, the most important are contained in what I call the Semantic Condition, which ensures that the MOOC remains a living system. The semantic condition contains four parts: autonomy, diversity, openness, and interactivity. The MOOC is assessed against each of these and a degree of compliance may be found.
The outcomes perspective looks at the MOOC as a knowing system. By that what I mean is that the MOOC should exhibit network properties on a macro scale - in other words, that we should be able to say things about the MOOC without reference to particular individuals in the MOOC. This is to treat the MOOC as an entity which perceives, or which learns, as a whole. These things are emergent properties, for example, emergent knowledge or emergent learning. Did the MOOC as a whole produce some new insight, or recognize some new phenomenon in its area of study?
MOOC success, in other words, is not individual success. We each have our own motivations for participating in a MOOC, and our own rewards, which may be more or less satisfied. But MOOC success emerges as a consequence of individual experiences. It is not a combination or a sum of those experiences - taking a poll won't tell us about them - but rather a result of how those experiences combined or meshed together.
This may not reflect what institutional funders want to hear. But my thinking and hope is that over th long term MOOCs will be self-sustaining, able to draw participants who can see the value of a MOOC for what it is, without needing to support narrow and specific commercial or personal learning objectives.