In my previous post I considered some difficulties I’m having in trying to figure out how to evaluate the effectiveness of cMOOCs. In this one I look at some of the things Stephen Downes has to say about this issue, and one research paper that uses his ideas as a lens through which to consider data from a cMOOC.
Stephen Downes on the properties of successful networks
This post by Stephen Downes (which was a response to a question I asked he and others via email) describes two ways of evaluating the success of a cMOOC through asking whether it fulfills the properties of successful networks. One could look at the “process conditions,” which for Downes are four: autonomy, diversity, openness, and interactivity. And/or, one could look at the outcomes of a cMOOC, which for Downes means looking at whether knowledge emerges from the MOOC as a whole, rather than just from one or more of its participants. I’ll look briefly at each of these ways of considering a cMOOC in what follows.
The four “process conditions” for a successful network are what Downes calls elsewhere a “semantic condition” that is required for a knowledge-generating network, a network that generates connective knowledge (for more on this, see longer articles here and here). This post discusses them succinctly yet with enough detail to give a sense of what they mean (the following list and quotes come from that post).
- Autonomy: The individuals in the network should be autonomous. One could ask, e.g.: “do people make their own decisions about goals and objectives? Do they choose their own software, their own learning outcomes?” This is important in order that the participants and connections form a unique organization rather than one determined from one or a few individuals, in which knowledge is transferred in as uniform a way as possible to all (this point is made more explicitly in the longer post attached here).
- Diversity: There must be a significant degree of diversity in the network for it to generate anything new. One could ask about the geographical locations of the individuals in the network, the languages spoken, etc., but also about whether they have different points of view on issues discussed, whether they have different connections to others (or does everyone tend to have similar connections), whether they use different tools and resources, and more.
- Openness: A network needs to be open to allow new information to flow in and thereby produce new knowledge. Openness in a community like a cMOOC could include the ease with which people can move into and out of the community/course, the ability to participate in different ways and to different degrees, the ability to easily communicate with each other. [Update June 14, 2013: Here Downes adds that openness also includes sharing content, both that from within the course to those outside of it, and that gained from outside (or created by oneself inside the course?) back into the course.]
- Interactivity: There should be interactivity in a network that allows for knowledge to emerge “from the communicative behaviour of the whole,” rather than from one or a few nodes.
To look at the success of a cMOOC from an “outcomes” perspective, you’d try to determine whether new knowledge emerged from the interactions in the community as a whole. This idea is a bit difficult for me to grasp, and I am having trouble understanding how I might determine if this sort of thing has occurred. I’ll look at one more thing here to try to figure this out.
Downes on the quality of MOOCs
Recently, Downes has written a post on the blog for the “MOOC quality project” that discusses how he thinks it might be possible to say whether a MOOC was successful or not, and in it he discusses the process conditions and outcomes further (to really get a good sense of his arguments, it’s best to read the longer version of this post, which is linked to the original).
Downes argues in the longer version that it doesn’t make sense to try to determine the purpose of MOOCs (qua MOOCs, by which I think he means as a category rather than as individual instances) based on “the reasons or motivations” of those offering or taking particular instances of them. This is because people may have varying reasons and motivations for creating and using MOOCs, which need not impinge on what makes for a good MOOC (just like people may use hammers in various ways–his example–that don’t impinge on whether a particular one hammer is a good hammer). Instead, he argues that we should look at “what a successful MOOC ought to produce as output, without reference to existing … usage.”
And what MOOCs ought to produce as output is “emergent knowledge,” which is
constituted by the organization of the network, rather than the content of any individual node in the network. A person working within such a network, on perceiving, being immersed in, or, again, recognizing, knowledge in the network thereby acquires similar (but personal) knowledge in the self.
Downes then puts this point differently, focusing on MOOCs:
[A] MOOC is a way of gathering people and having them interact, each from their own individual perspective or point of view, in such a way that the structure of the interactions produces new knowledge, that is, knowledge that was not present in any of the individual communications, but is produced as a result of the totality of the communications, in such a way that participants can through participation and immersion in this environment develop in their selves new (and typically unexpected) knowledge relevant to the domain.
He then argues that the four process conditions discussed previously usually tend to produce this sort of emergent knowledge as a result, in the ways suggested in the above list. But, properties like diversity and openness are rather like abstract concepts such as love or justice in that they are not easily “counted” but rather need to be “recognized”: “A variety of factors–not just number, but context, placement, relevance and salience–come into play (that is why we need neural networks (aka., people) to perceive them and can’t simply use machines to count them.”
So far, so good; one might think it possible to come up with a way to evaluate a MOOC by looking at these four process conditions, and then assume that if they are in place, emergent knowledge is at least more likely to result (though it may not always do so). It would not be easy to figure out how to determine if these conditions are met, but one could come up with some ways to do so that could be justified pretty well, I think (even though there might be multiple ways to do so).
MOOCs as a language
But Downes states that while such an exercise may be useful when designing a course, it is less so when evaluating one after the fact–I’m not sure why this should be the case, though. He states that looking at the various parts of a course in terms of these four conditions (such as the online platform, the content/guest speakers, and more) could easily become endless–one could look at many, many aspects of a MOOC this way. But I don’t see why that would be more problematic in evaluating a course than in designing one.
Instead, Downes suggests we take a different tack in measuring success of MOOCs. He suggests we think of MOOCs as a language, “and the course design (in all its aspects) therefore as an expression in that language.” This is meant to take us away from the idea of using the four process conditions above as a kind of rubric or checklist in a mechanical way. The point rather is for someone who is already fluent in either MOOC design or the topic(s) being addressed in a MOOC to be able to look at the MOOC and the four conditions and “recognize” whether it has been successful or not. Downes states that “the bulk of expertise in a language–or a trade, science or skill–isn’t in knowing the parts, but in fluency and recognition, cumulating in the (almost) intuitive understanding (‘expertise’, as Dreyfus and Dreyfus would argue)” (here Downes refers to: http://www.sld.demon.co.uk/dreyfus.pdf).
So I think the idea here is that once one is fluent in the language of MOOCs or the “domain or discipline” of the topics they are about, one should be able to read and understand the expression in that language that is the course design, and to determine the quality of the MOOC by using the four conditions as a kind of “aid” rather than “checklist”. But to be quite honest, I am still not sure what it means, exactly, to use them as an “aid.” And this process suggests relying on those who have developed some degree of expertise in MOOCs to be able to make the judgment, thereby making the decision of successful vs. unsuccessful MOOCs come only from a set of experts.
Perhaps this could make sense, if we think of MOOCs like the product of some artisanal craft, like swordmaking–maybe it really is only the experts who can determine their quality, because perhaps there is no way to set out in a list of necessary and sufficient conditions what is needed for a successful MOOC, like it’s difficult (or impossible) to do for a high-quality sword (I’m just guessing on that one). Perhaps there are so many different possible ways of having a high quality MOOC/sword, with some aspects being linked to individual variations such that it’s impossible to describe each possible variation and what aspects of quality would be required for that particular variation. It may be that no one can possibly know in advance what all the possible variations of a successful MOOC/sword are, but that these can be recognized later.
But I’m not yet convinced that must be the case for MOOCs, at least not from this short essay. And I expect I would benefit from a closer reading of Downes’ other work, which might help me see why he’s going in this direction here. It would also help me see why he thinks the process conditions for a knowledge-generating network should be the ones he suggests.
Using Downes’ framework to evaluate the effectiveness of a cMOOC
This is a bit premature, as I admit I don’t understand it in its entirety, but I want to put out a few preliminary ideas. I’m leaving aside, for the moment, the idea of MOOCs as a language until I figure out more precisely why he thinks we should look at them that way, and then decide if I agree. I’m also leaving aside for the moment the question of whether I think the process conditions he suggests are really the right ones–I haven’t evaluated them or the reasons behind them and thus can’t say one way or the other at this point.
The four process conditions
One would have to figure out exactly how to define Autonomy, Diversity, and Openness, which is no easy task, but it seems possible to come to a justifiable (though not final or probably perfect) outline of what those mean, considering what might make for a knowledge-generating network. It might be a long and difficult process to do so, but at least possible, I think. Then, it would be fairly straightforward to devise a manageable (and only ever partial) list of things one could ask about, measure, humanly “recognize” (in the sense of not using a checklist mechanically…though again, I’m not entirely sure what that means) to see if a particular cMOOC fit these three criteria. Again, I have no idea how to do any of this right now, but I think it could be done.
But I am still unsure about the final one: interactivity. This is because it’s not just a matter of people interacting with each other; rather, Downes emphasizes that what is needed is interaction that allows for emergent knowledge. So to figure this one out, one already needs to understand what emergent knowledge looks like and how to recognize if it has happened. I understand the idea of emergent knowledge in an abstract sense, but it’s hard to know how I would figure out if some knowledge had emerged from the communicative interactions of a community rather than from a particular node or nodes. How would I tell if, as quoted above, “the structure of the interactions produce[d] new knowledge, that is, knowledge that was not present in any of the individual communications, but [was] produced as a result of the totality of the communications”? Or, to take another quote from the longer version of the post Downes did for the “MOOC quality project”, how would I know if “new learning occur[red] as a result of this connectedness and interactivity, it emerge[d] from the network as a whole, rather than being transmitted or distributed by one or a few more powerful members”?
I honestly am having a hard time figuring out where/how to look for knowledge that wasn’t present in any of the individual communications, but emerges from the totality of them. But I think part of the problem here is that I don’t understand enough about Downes’ view of connectivism and connectivist knowledge. I knew I should take a closer look at connectivism before trying to tackle the question of evaluating cMOOCs! Guess I’ll have to come back to this after doing a post or two on Downes’ view of connectivism & connective knowledge.
So clearly I have a long way to go to understand exactly what Downes is suggesting and why, before I can even decide if this would be a good framework for evaluating a cMOOC.
In a later post I will look at two research papers that look at cMOOCs through the lens of Downes’ four process conditions, to see how they have interpreted and used these.
I welcome comments on anything I’ve said here–anything I’ve gotten wrong, or any suggestions on what I’m still confused about?