The Study, and Other Stuff

There are three separate threads in Siemens's response to my last post, all of which are fascinating:

• The thread concerning whether or not the study he published was bad,

• The thread examining the question of whether universities can be a valuable force for social equity, and

• My own experiences of the university system.

Though the latter two threads are of endless interest, I'd really rather only focus on the first, for today.


Whether or not the study he published was bad

Siemens writes, "Stephen expands on his primary concerns which are about educational research in general." Let me be clear: I was making this statement about this study in particular. That's why I cited work from the study itself. Yes, I believe that educational work in general is pretty poor. But my focus was on this particular example.

I think he agrees with me, in part:

Educational research is often poorly done. Research in social systems is difficult to reduce to a set of variables and relationships between those variables. Where we have large amounts of data, learning analytics can provide insight, but often require greater contextual and qualitative data. ... The US Department of Education has a clear articulation of what they will count as evidence for grants. It’s a bit depressing, actually, a utopia for RCTs (Randomized Controlled Trials).

And he says:

Stephen then makes an important point and one that needs to be considered that the meta-studies that we used are “hopelessly biased in favour of the traditional model of education as practiced in the classrooms where the original studies took place.” This is a significant challenge. How do we prepare for digital universities when we are largely duplicating classrooms? Where is the actual innovation? (I’d argue much of it can be fore in things like cmoocs and other technologies that we address in chapter 5 of the report). Jon Dron largely agrees with Stephen and suggests that a core problem exists in the report in that it is a “view from the inside, not from above.”

So, from this, it appears that he agrees with my criticisms.

He nonetheless persists with his defense, focusing on the fifth paper in the study, first suggesting I don't find a lot to disagree with about it, and second, suggesting it is a vehicle for a conversation between two versions of myself. He also finds fault with some other criticisms:

The names listed were advisors on the MOOC Research Initiative – i.e. they provided comments and feedback on the timelines and methods. They didn’t select the papers. The actual peer review process included a much broader list, some from within the academy and some from the outside. 

Who selected the review committee? Who are the people 'from the outside' that were on it? Here's the best we have on the review process itself. Here are the project reports. All of this was set in motion by the committee I named in my previous post. If there's another list of names of people who were responsible for the outcome, they should be named. Otherwise, the people named are the people responsible. You can't name a list of names and then say it wasn't them.

In his defense of the fifth paper (he seems not to defend the first four studies, the 'histories', at all) he also writes:

In my previous post, I stated that we didn’t add to citations. We analyzed those that were listed in the papers that others submitted to MRI. Our analysis indicated that popular media influenced the MOOC conversation and the citations used by those who submitted to the grant.

I recognize this. What I am is saying is that it seems to me that the 28 winners of a major education research grant competition would have demonstrated more depth of understanding that is apparent from the summary study that resulted. Maybe I should not have expected more from what was essentially an automated and quantitative analysis of the papers (because there are individually some bright spots). But when we look at the citations - which is essentially what we were provided - the results overall are not reassuring.

That's it for Siemens's defense of the study. The core of my criticism, which is addressed mostly at the first four chapters, s is not addressed. Let me reiterate them here:

• They all have very small sample sizes, usually less than 50 people, with a maximum size less than 200 people
• The people studied are exclusively university students enrolled in a traditional university course
• The method being studies is almost exclusively the lecture method
• The outcomes are assessed almost exclusively in the form of test results
• Although many are 'controlled' studies, most are not actually controlled for "potential confounders"
• All these criticisms apply if you think this is the appropriate sort of study to measure educational effectiveness, which I do not.

I would not like to add that my criticisms are reinforced by two additional authors.

Although Jon Dron says "as such reports go, I think it is a good one," he writes:

For the most part, this report is a review of the history and current state of online/distance/blended learning in formal education. This is in keeping with the title, but not with the ultimate thrust of at least a few of the findings. That does rather stifle the potential for really getting under the skin of the problem. It's a view from the inside, not from above. 

And additionally, George Veletsianos writes,

One of Downes  criticisms is the following: “the studies are conducted by people without a background in education.” This finding lends some support to his claim, though a lot of the research on MOOCs is from people affiliated with education, but to support that claim further one could examine the content of this papers and identify whether an educational theory is guiding their investigations.

I don't think it matters whether the investigation is informed by an educational theory - all I care about is that the studies contribute in a useful, relevant and credible way to the field.

Finally, Siemens says, "The appeal to evidence is to essentially state that opinions alone are not sufficient."

It can be allowed that Siemens's use of "we" in the Chronicle article "is about the academy’s embrace of MOOCs." But as I pointed out, there's no mistaking his suggestion that the people outside the academy, the Alt-ac people, do not rely on evidence. This is what he says when he says, "Another approach, and one that I see as complimentary and not competitive, is to emphasize research and evidence."

I have never suggested that opinion alone is sufficient, and never would. But he has to cease characterizing the alternatives as not evidence based. Because I believe the opposite. I believe that the controlled trials offered in the study misrepresent what little evidence they provide, and I believe that the alternative approaches offer substantially more evidence than is allowed.


Siemens says, "While Stephen says our evidence is poor, he doesn’t provide what he feels is better evidence." I did once author a Guide to the Logical Fallacies, where I discuss the statistical problems. I've also talked about the same issue of evidence as it related to public policy. I've talked about research methodologies a number of times. And just the other day, I linked to a study I felt did pass muster (and indeed, over the years, I've linked to lots of things that I felt met the appropriate standards of research and evidence). And the body of my work, grounded in practical application and observation, stands as an example of what I feel constitutes "better evidence."

The Other Stuff

It's late and I don't want to longer on the off-topic stuff. But I also want to address a few things.

It's true that I am not a fan of universities and do not feel they support our common objective of " an equitable society with opportunities for all individuals to make the lives that they want without institutions (and faculty in this case) blocking the realization of those dreams."

This does not mean that I want to see them eliminated. And (contrary to Sebastian Thrun) I expect their numbers will multiply exponentially in the future.

But they need to be reformed, and they need to be brought around to the idea that social and economic equity are important. Because as it stands, they are one of the largest bastions in society standing against that idea. Here are a few of the ways:

- universities foster the perpetuation of a social elite, especially through exclusive institutions (Harvard, Yale, etc), legacy admissions, and perpetuation of a private social society consisting pretty much only of the one-percent

- universities bleed those outside the upper classes by consistently responding to society's demand for access with higher and higher tuition fees

- universities have fostered the creation of a low-paid academic underclass in order to support the students that pay these higher fees, and resist any suggestion that they should be fairly compensated, and actively resist unionization

- universities and professors continue to contribute to mechanisms which keep academic research behind expensive paywalls - indeed, they are so indifferent to these costs that they must be required by mandates and laws to open access to their research

- private universities operate tax-free, raise substantial endowment funds (sometimes in the billions), yet always plead poverty, and are typically the prime recipient of funding provided by governments and foundations attempting to support projects leading to the betterment of social and economic conditions

- they then waste that money, and a lot of other money, padding their own resumes and producing research such as the body of work I find myself criticizing today

Yes, perhaps universities could act as a force that promotes social and economic equity. They certainly have the talent and resources. But they don't, they don't want to, and they resist any attempt to make them do it.

It is true that I was badly treated by my PhD committee. But this is not a case of "today affirming that the Stephen in front of the phd committee made the right decision – that there are multiple paths to research, that institutions can be circumvented and that individuals, in a networked age, have control and autonomy." Why not? A couple of reasons:


On the idea that, individuals, in a networked age, (should) have control and autonomy: I have always believed that. I believed that long before I ever stood before a PhD committee.

On the idea that "the Stephen that today has exceeded the impact of members on that committee through blogging, his newsletter, presentations, and software writing."This may or may not be true. But I have never believed that I have been more influential because I have worked outside of academia.

I have been influential despite being outside academia. I have been influential despite not having a professor's wages, the support of grad students, a year off every seven, tenure, funding from foundations, grants and agencies, book contracts, and the rest. No university in the world would ever hire me, because they consider me unqualified. I don't regard any of this really as an upside.

Because that's what academia does. It wields huge sums of money and the support to achieve certain social and economic outcomes. I just wish it was wielding this power for good, rather than indifference. But I don't think it ever will.