Teaching & Learning, and SoTL, in Philosophy
I gave a presentation at the Open Education Conference 2013 in Park City, Utah on Nov. 8, 2013. See my previous post for video, slides and bibliography.
I also wanted to see what people were saying about it during the presentation, in case there were some ideas there that are useful for my continuing research into this issue (and there were!). So I made a Storify story. Here’s the link to it on Storify if you’d rather see it there.
Given Nov. 8, 2013, at the Open Education Conference 2013 at Park City, Utah.
Here is the video recording. I had only 25 minutes to present, and I was late starting because I was messing with my computer, trying to get it to show me “presenter mode” while it showed the slides on the screen so I could see my notes. Then I tried to see my notes on my phone. Then I gave up on my notes and just winged it! (I was using Keynote rather than PowerPoint, and I’ve never tried to use presenter mode before…the problem was that I couldn’t print out my notes because the printer in the “business centre” of the hotel was out of order!)
Here are the slides, which are licensed CC-BY so you can use any part of them if you want. Again, these were in Apple Keynote, and when I exported to PowerPoint some of the colours, fonts and alignments got messed up a bit.
When I get a free half a day (probably in December) I’ll write up a post in which I explain my argument in this presentation, including the slides at the end I didn’t get to!
Update Feb. 2015: Well, obviously I never wrote this up. Which is too bad, because now it’s been quite awhile and it would take me a long time to try to do so. I do plan to return to this research at some point (perhaps in the Summer of 2015), and see what else has been published in the meantime. And who knows what kind of open online course models there will be by then?!
Bibliography
Things either cited on the slides or quoted from in the presentation (at least, the original version as I wrote it, not the shortened one given in the video!)
Ahn, J., Weng, C., & Butler, B. S. (2013). The Dynamics of Open, Peer-to-Peer Learning: What Factors Influence Participation in the P2P University? (pp. 3098–3107). IEEE. doi: 10.1109/HICSS.2013.515
Cormier, D. (2010a). Knowledge in a MOOC – YouTube. Retrieved from https://www.youtube.com/watch?v=bWKdhzSAAG0
Cormier, D., & Siemens, G. (2010b). Through the Open Door: Open Courses as Research, Learning, and Engagement. Educause Review, 45(4), 30–39. Retrieved from http://www.educause.edu/ero/article/through-open-door-open-courses-research-learning-and-engagement
Downes, S. (2007, February 3). What Connectivism Is. Half an Hour. Retrieved from http://halfanhour.blogspot.com.au/2007/02/what-connectivism-is.html
Downes, S. (2009, February 24). Connectivist Dynamics in Communities. Half an Hour. Retrieved from http://halfanhour.blogspot.co.uk/2009/02/connectivist-dynamics-in-communities.html
Downes, S. (2013a). Supporting a Distributed Online Course ~ Stephen’s Web. Presented at the Information Technology Based Higher Education and Training ITHET 2013, Antalya, Turkey. Retrieved from http://www.downes.ca/presentation/327
Downes, S. (2013b). The Quality of Massive Open Online Courses. MOOC Quality Project. Retrieved from http://mooc.efquel.org/week-2-the-quality-of-massive-open-online-courses-by-stephen-downes/ A longer version of this post can be found here: http://cdn.efquel.org/wp-content/blogs.dir/7/files/2013/05/week2-The-quality-of-massive-open-online-courses-StephenDownes.pdf
Fournier, H., Kop, R., & Sitlia, H. (2011). The Value of Learning Analytics to Networked Learning on a Personal Learning Environment. Presented at the 1st International Conference Learning Analytics and Knowledge, Banff, Alberta. Retrieved from http://nparc.cisti.nrc.ca/npsi/ctrl?action=shwart&index=an&req=18150452&lang=en
Kop, R. (2011). The challenges to connectivist learning on open online networks: Learning experiences during a massive open online course. The International Review of Research in Open and Distance Learning, 12(3), 19–38. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/882
Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy to support human beings? Participant support on massive open online courses. The International Review of Research in Open and Distance Learning, 12(7), 74–93. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1041
Lane, L. M. (2013). An Open, Online Class to Prepare Faculty to Teach Online. Journal of Educators Online, 10(1), n1. Retrieved from http://www.thejeo.com/Archives/Volume10Number1/Lane.pdf
Mackness, J., Mak, S., & Williams, R. (2010). The ideals and reality of participating in a MOOC. In Proceedings of the 7th International Conference on Networked Learning 2010 (pp. 266–275). University of Lancaster. Retrieved from http://eprints.port.ac.uk/5605/
Milligan, C., Littlejohn, A., & Margaryan, A. (2013). Patterns of Engagement in Connectivist MOOCs. Journal of Online Teaching and Learning, 9(2). Retrieved from http://jolt.merlot.org/vol9no2/milligan_0613.htm
Siemens, G. (2006, November 12). Connectivism: Learning Theory or Pastime for the Self-Amused? elearnspace. Retrieved from http://www.elearnspace.org/Articles/connectivism_self-amused.htm
Siemens, G. (2008, August 6). What is the unique idea in Connectivism? « Connectivism. Connectivism. Retrieved from http://www.connectivism.ca/?p=116
Waite, M., Mackness, J., Roberts, G., & Lovegrove, E. (2013). Liminal Participants and Skilled Orienteers: Learner Participation in a MOOC for New Lecturers. Journal of Online Teaching and Learning, 9(2). Retrieved from http://jolt.merlot.org/vol9no2/waite_0613.htm
Williams, R., Karousou, R., & Mackness, J. (2011). Emergent learning and learning ecologies in Web 2.0. The International Review of Research in Open and Distance Learning, 12(3), 39–59. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/883
Williams, R. T., Mackness, J., & Gumtau, S. (2012). Footprints of emergence. The International Review of Research in Open and Distance Learning, 13(4), 49–90. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1267
This is the third in a series of posts on a research project I’m developing on evaluating cMOOCs. The first can be found here, and the second here. In this post I consider an article that uses Downes’ four “process conditions” for a knowledge-generating network to evaluate a cMOOC. In a later post I’ll consider another article that takes a somewhat critical look at these four conditions as applied to cMOOCs.
Mackness, J., Mak, S., & Williams, R. (2010). The ideals and reality of participating in a MOOC. In Proceedings of the 7th International Conference on Networked Learning 2010 (pp. 266–275). Retrieved from http://eprints.port.ac.uk/5605/
Connexion, Flickr photo by tangi_bertin, licensed CC-BY
In this article, Mackness et al. report findings from interviews of participants in the CCK08 MOOC (Connectivism and Connective Knowledge 2008; see here for a 2011 version of this course) insofar as these relate to Downes’ four process conditions for a knowledge-generating network: autonomy, diversity, openness, interactivity. In other words, they wanted to see if these conditions were met in CCK08, according to the participants. To best understand these results, if you’re not familiar with Downes’ work, it may be helpful to read an earlier post of mine that addresses and tries to explain these conditions.
Specifically, the researchers asked: “To what extent were autonomy, diversity, openness and connectedness/interactivity a reality for participants in the CCK08 MOOC and how much they were affected by the course design?” (271). They concluded that, in this particular course at least, there were difficulties with all of these factors.
Data
Data for this study came from 22 responses by participants (including instructors) to email interview questions (out of 58 who had self-selected, on a previous survey sent to 301 participants, to be interviewed). Unfortunately, the interview questions are not provided in the paper, so it’s hard to tell what the respondents were responding to. I find it helpful to see the questions so as to better understand the responses given, and be able to undertake a critical review of the interpretation of those responses given in an article.
Results
Autonomy
The researchers note that most respondents valued autonomy in a learning environment: “Overall, 59% of interview respondents (13/22) rated the importance of learner autonomy at 9 or 10 on a scale of 1-10 (1 = low; 10 = high)” (269). Unfortunately, I can’t tell if this means they valued the kind of autonomy they experienced in that particular course, or whether they valued the general idea of learner autonomy in an abstract way (but how was it defined?). Here is one place, for example, where providing the question asked would help readers understand the results.
Mackness et al. then argue that nevertheless, some participants (but how many out of the 22?) found the experience of autonomy in CCK08 to be problematic. The researchers provided quotes from two participants stating that they would have preferred more structure and guidance, and one course instructor who reported that learner autonomy led to some frustration that what s/he was trying to say or do in the course was not always “resonating with participants” (269).
The authors also provide a quote from a course participant who said they loved being able to work outside of assessment guidelines, but then comment on that statement by saying that “autonomy was equated with lack of assessment”–perhaps, but not necessarily (maybe they could get good feedback from peers, for example? Or maybe the instructors could still assess something outside of the guidelines? I don’t know, but the statement doesn’t seem to mesh, by itself, with the interpretation). Plus, the respondent saw this as a positive thing, whereas the rhetorical aspects of the interpretation suggest it was a negative, a difficulty with autonomy. I’m not seeing that.
The researchers conclude that the degree of learner autonomy in the course was affected by the following:
levels of fluency in English, the ‘expertise divide’, assessment for credit participants, personal learning styles, personal sense of identity and the power exerted, either implicitly or explicitly, by instructors through their communications, status and reputation, or by participants themselves….” (271)
In addition, there were reports of some “trolling” behaviour on the forums, which led some participants to “retreat to their blogs, effectively reducing their autonomy” (271). The authors point out that some constraint on autonomy in the forums through discouraging or shutting down such behaviour may have actually promoted autonomy amongst more learners.
Diversity
The researchers note that learner diversity was certainly present in the course, including diversity in geography, language, age, and background. They give examples of diversity “reflected in the learning preferences, individual needs and choices expressed by interview respondents” (269).
However, diversity was also a problem in at least one respect, namely that not all learners had the “skills or disposition needed to learn successfully, or to become autonomous learners in a MOOC” (271). This is not so much of a problem if there is significant scaffolding, such as support for participants’ “wayfinding in large online networks,” but CCK08 was instead designed to have “minimal instructor intervention” (271). In addition, in order to promote sharing in a network like a cMOOC, there needs to be a certain amount of trust built up, the authors point out; and the more large and diverse the network, the more work may need to be done to help participants build that trust.
Openness
Into the Great Wide Open, Flickr photo by Claudio Matsuoka, licensed CC-BY
CCk08 was available, for free, to anyone who wanted to participate (without receiving any university or other credits), so long as they had a reliable web connection. The interview data suggests that participants interpreted “openness” differently: some felt they should (and did) share their work with others (thus interpreting openness as involving sharing one’s work), some worked mostly alone and did not do much or any sharing–thereby interpreting openness, the author suggest, merely as the idea that the course was open for anyone with a reliable web connection to participate in. The authors seem to be arguing here that these differing conceptions of openness are problematic because there was an “implicit assumption in the course was that participants would be willing or ready to give and receive information, knowledge, opinions and ideas; in other words to share freely” (270), but that not everyone got that message. They point to a low rate of active participation: only 14% of the total enrolled participants (270).
They also note that amongst participants there was no “common understanding of openness as a characteristic of connectivism” (270), implying that there should have been. But I wonder if conscious understanding of openness, and the ability to express that as a clear concept, is necessary for a successful connectivist course. This is just a question at this point–I haven’t thought it through carefully. I would at least have liked to have seen more on why that should be considered a problem, as well as whether the respondents were asked specifically for their views of openness. The responses given in this section of the paper don’t refer to openness at all, making me think perhaps the researchers interpreted understandings of openness from one or more of the other things respondents said. That’s not a problem by itself, of course, but one might have gotten different answers if one had asked them their views of openness directly, and answers that might have been therefore more relevant to concluding whether or not participants shared a common understanding of openness.
Finally, Mackness et al. argue that some of the barriers noted above also led to problems in regard to participants’ willingness to openly communicate and share work with others: this can be “compromised by lack of clarity about the purpose and nature of the course, lack of moderation in the discussion forums, which would be expected on a traditional course, and the constraints (already discussed in relation to autonomy and diversity) under which participants worked” (272).
Interactivity
There were significant opportunities for interaction, for connecting with others, but the authors note that what is most important is not whether people did connect with others (and how much) as what these connections made possible. Respondents noted some important barriers to connecting as well as problems that meant some of the interactions did not yield useful benefits. As noted above, some participants pointed to “trolling” behaviour on the forums, and one said there were some “patronising” posts as well–which, the respondent said, likely led some participants to disengage from that mode of connection. Another respondent noted differences in expertise levels that led him/her to disengage when s/he could no longer “understand the issues being discussed” (271).
The researchers conclude that connectivity alone is not sufficient for effective interactivity–which of course makes sense–and that the degree of effective interactivity in CCK08 was not as great as it might have been with more moderation by instructors. However, the size of the course made this unfeasible (272).
One thing I would have liked to have seen in this analysis of “interactivity” is what Downes focuses on for this condition, namely the idea that the kind of interactivity needed is that which promotes emergent knowledge–knowledge that emerges from the interactions of the network as a whole, rather than from individual nodes (explained by Downes here and here, the first of which the authors themselves cite). This is partly because if they used Downes’ framework, it would make sense to evaluate the course with the specifics of what he means by “interactivity.” It’s also partly because I just really want to see how one might try to evaluate that form of interactivity.
Conclusion
Mackness et al. conclude that
some constraints and moderation exercised by instructors and/or learners may be necessary for effective learning in a course such as CCK08. These constraints might include light touch moderation to reduce confusion, or firm intervention to prevent negative behaviours which impede learning in the network, and explicit communication of what is unacceptable, to ensure the ‘safety’ of learners. (272)
Though, at the same time, they point to the small size of their sample, and the need for further studies of these sorts of courses to validate their findings.
That makes sense to me, from my unstudied perspective of someone who has participated in a few large and one small-ish open online courses, one of which seemed modeled to some degree along connectivist lines (ETMOOC). There was some significant scaffolding in ETMOOC, through starting off with discussions of connected learning and help with blogging and commenting on blogs. There wasn’t clear evidence of moderating discussions from the course collaborators (several people collaborated on each two-week topic, acting in the role of “instructors” for a brief time), except insofar as some of the course collaborators were very actively present on Twitter and in commenting on others’ blogs, being sure to tweet or retweet or bookmark to Diigo or post to Google+ especially helpful or thought-provoking things. We didn’t have any trolling behaviour that I was aware of, and we also didn’t have a discussion forum. But IF there were problems in the Google+ groups or in Twitter chats, I would have hoped one or more of the collaborators would have actively worked to address them (and I think they would have, though of course since it didn’t happen (to my knowledge) I can’t be certain).
Some further thoughts
If one decides that Downes’ framework is the right one to use for evaluating an open online course like a cMOOC (which I haven’t decided yet; I still need to look more carefully at his arguments for it), it would make sense to unpack the four conditions more carefully and collect participants’ views on whether those specific ways of thinking about autonomy, diversity, openness and interactivity were manifested in the course. The discussion of these four conditions is at times rather vague here. What, more specifically, does learner “autonomy” mean, for example? Even if they don’t want to use Downes’ own views of autonomy, it would be helpful to specify what conception of autonomy they’re working with. I’ve also noted a similar point about interactivity, about which the discussion in the paper is also somewhat vague–what sort of interactivity would have indicated success, exactly, beyond just participants communicating with each other on blogs or forums?
I find it interesting that in his most recent writing on the topic of evaluating cMOOCs (see the longer version attached to this post, and my discussion of this point here (and the helpful comments I’ve gotten on that post!)), Downes argues that it should be some kind of expert in cMOOCs or in one of the fields/topics they cover that evaluates their quality, while here the authors looked to the participants’ experiences. Interesting, because it makes sense to me to actually focus on the experiences of the participants rather than to ask someone who may or may not have taken the course. That is, if one wants to find out if the course was effective for participants.
Still, I can see how some aspects of these conditions might be measured without looking at what participants experienced, or at least in other ways in addition to gathering participants’ subjective evaluations. The degree to which the course is “open,” for example, might have some elements that could be measured beyond or in addition to what participants themselves thought. Insofar as openness involves the course being open to anyone with a reliable internet connection to participate, without cost, and the ability to move into and out of the course easily as participants choose, that could be partly a matter of looking at the design and platform of the course itself, as well as participants’ evaluations of how easy it was to get into and out of the course. If openness also involves the sharing of one’s work, one could look to see how much of that was actually done, as well as ask participants about what they shared, why, and how (and what they did not, and why).
I just find it puzzling that in that recent post Downes doesn’t talk about asking participants about their experiences in a cMOOC at all. I’m not sure why.
[I just read a recent comment on an earlier post, which I haven’t replied to yet, which discusses exactly this point–it makes no sense to leave out student experiences. Should have read and replied to that before finalizing this post!]
In my previous post I considered some difficulties I’m having in trying to figure out how to evaluate the effectiveness of cMOOCs. In this one I look at some of the things Stephen Downes has to say about this issue, and one research paper that uses his ideas as a lens through which to consider data from a cMOOC.
Stephen Downes on the properties of successful networks
This post by Stephen Downes (which was a response to a question I asked he and others via email) describes two ways of evaluating the success of a cMOOC through asking whether it fulfills the properties of successful networks. One could look at the “process conditions,” which for Downes are four: autonomy, diversity, openness, and interactivity. And/or, one could look at the outcomes of a cMOOC, which for Downes means looking at whether knowledge emerges from the MOOC as a whole, rather than just from one or more of its participants. I’ll look briefly at each of these ways of considering a cMOOC in what follows.
Social Network in a Course by Hans Põldoja. Licensed CC-BY
The four “process conditions” for a successful network are what Downes calls elsewhere a “semantic condition” that is required for a knowledge-generating network, a network that generates connective knowledge (for more on this, see longer articles here and here). This post discusses them succinctly yet with enough detail to give a sense of what they mean (the following list and quotes come from that post).
To look at the success of a cMOOC from an “outcomes” perspective, you’d try to determine whether new knowledge emerged from the interactions in the community as a whole. This idea is a bit difficult for me to grasp, and I am having trouble understanding how I might determine if this sort of thing has occurred. I’ll look at one more thing here to try to figure this out.
Downes on the quality of MOOCs
Recently, Downes has written a post on the blog for the “MOOC quality project” that discusses how he thinks it might be possible to say whether a MOOC was successful or not, and in it he discusses the process conditions and outcomes further (to really get a good sense of his arguments, it’s best to read the longer version of this post, which is linked to the original).
Downes argues in the longer version that it doesn’t make sense to try to determine the purpose of MOOCs (qua MOOCs, by which I think he means as a category rather than as individual instances) based on “the reasons or motivations” of those offering or taking particular instances of them. This is because people may have varying reasons and motivations for creating and using MOOCs, which need not impinge on what makes for a good MOOC (just like people may use hammers in various ways–his example–that don’t impinge on whether a particular one hammer is a good hammer). Instead, he argues that we should look at “what a successful MOOC ought to produce as output, without reference to existing … usage.”
And what MOOCs ought to produce as output is “emergent knowledge,” which is
constituted by the organization of the network, rather than the content of any individual node in the network. A person working within such a network, on perceiving, being immersed in, or, again, recognizing, knowledge in the network thereby acquires similar (but personal) knowledge in the self.
Downes then puts this point differently, focusing on MOOCs:
[A] MOOC is a way of gathering people and having them interact, each from their own individual perspective or point of view, in such a way that the structure of the interactions produces new knowledge, that is, knowledge that was not present in any of the individual communications, but is produced as a result of the totality of the communications, in such a way that participants can through participation and immersion in this environment develop in their selves new (and typically unexpected) knowledge relevant to the domain.
He then argues that the four process conditions discussed previously usually tend to produce this sort of emergent knowledge as a result, in the ways suggested in the above list. But, properties like diversity and openness are rather like abstract concepts such as love or justice in that they are not easily “counted” but rather need to be “recognized”: “A variety of factors–not just number, but context, placement, relevance and salience–come into play (that is why we need neural networks (aka., people) to perceive them and can’t simply use machines to count them.”
So far, so good; one might think it possible to come up with a way to evaluate a MOOC by looking at these four process conditions, and then assume that if they are in place, emergent knowledge is at least more likely to result (though it may not always do so). It would not be easy to figure out how to determine if these conditions are met, but one could come up with some ways to do so that could be justified pretty well, I think (even though there might be multiple ways to do so).
MOOCs as a language
But Downes states that while such an exercise may be useful when designing a course, it is less so when evaluating one after the fact–I’m not sure why this should be the case, though. He states that looking at the various parts of a course in terms of these four conditions (such as the online platform, the content/guest speakers, and more) could easily become endless–one could look at many, many aspects of a MOOC this way. But I don’t see why that would be more problematic in evaluating a course than in designing one.
Instead, Downes suggests we take a different tack in measuring success of MOOCs. He suggests we think of MOOCs as a language, “and the course design (in all its aspects) therefore as an expression in that language.” This is meant to take us away from the idea of using the four process conditions above as a kind of rubric or checklist in a mechanical way. The point rather is for someone who is already fluent in either MOOC design or the topic(s) being addressed in a MOOC to be able to look at the MOOC and the four conditions and “recognize” whether it has been successful or not. Downes states that “the bulk of expertise in a language–or a trade, science or skill–isn’t in knowing the parts, but in fluency and recognition, cumulating in the (almost) intuitive understanding (‘expertise’, as Dreyfus and Dreyfus would argue)” (here Downes refers to: http://www.sld.demon.co.uk/dreyfus.pdf).
So I think the idea here is that once one is fluent in the language of MOOCs or the “domain or discipline” of the topics they are about, one should be able to read and understand the expression in that language that is the course design, and to determine the quality of the MOOC by using the four conditions as a kind of “aid” rather than “checklist”. But to be quite honest, I am still not sure what it means, exactly, to use them as an “aid.” And this process suggests relying on those who have developed some degree of expertise in MOOCs to be able to make the judgment, thereby making the decision of successful vs. unsuccessful MOOCs come only from a set of experts.
Perhaps this could make sense, if we think of MOOCs like the product of some artisanal craft, like swordmaking–maybe it really is only the experts who can determine their quality, because perhaps there is no way to set out in a list of necessary and sufficient conditions what is needed for a successful MOOC, like it’s difficult (or impossible) to do for a high-quality sword (I’m just guessing on that one). Perhaps there are so many different possible ways of having a high quality MOOC/sword, with some aspects being linked to individual variations such that it’s impossible to describe each possible variation and what aspects of quality would be required for that particular variation. It may be that no one can possibly know in advance what all the possible variations of a successful MOOC/sword are, but that these can be recognized later.
But I’m not yet convinced that must be the case for MOOCs, at least not from this short essay. And I expect I would benefit from a closer reading of Downes’ other work, which might help me see why he’s going in this direction here. It would also help me see why he thinks the process conditions for a knowledge-generating network should be the ones he suggests.
Using Downes’ framework to evaluate the effectiveness of a cMOOC
This is a bit premature, as I admit I don’t understand it in its entirety, but I want to put out a few preliminary ideas. I’m leaving aside, for the moment, the idea of MOOCs as a language until I figure out more precisely why he thinks we should look at them that way, and then decide if I agree. I’m also leaving aside for the moment the question of whether I think the process conditions he suggests are really the right ones–I haven’t evaluated them or the reasons behind them and thus can’t say one way or the other at this point.
The four process conditions
One would have to figure out exactly how to define Autonomy, Diversity, and Openness, which is no easy task, but it seems possible to come to a justifiable (though not final or probably perfect) outline of what those mean, considering what might make for a knowledge-generating network. It might be a long and difficult process to do so, but at least possible, I think. Then, it would be fairly straightforward to devise a manageable (and only ever partial) list of things one could ask about, measure, humanly “recognize” (in the sense of not using a checklist mechanically…though again, I’m not entirely sure what that means) to see if a particular cMOOC fit these three criteria. Again, I have no idea how to do any of this right now, but I think it could be done.
But I am still unsure about the final one: interactivity. This is because it’s not just a matter of people interacting with each other; rather, Downes emphasizes that what is needed is interaction that allows for emergent knowledge. So to figure this one out, one already needs to understand what emergent knowledge looks like and how to recognize if it has happened. I understand the idea of emergent knowledge in an abstract sense, but it’s hard to know how I would figure out if some knowledge had emerged from the communicative interactions of a community rather than from a particular node or nodes. How would I tell if, as quoted above, “the structure of the interactions produce[d] new knowledge, that is, knowledge that was not present in any of the individual communications, but [was] produced as a result of the totality of the communications”? Or, to take another quote from the longer version of the post Downes did for the “MOOC quality project”, how would I know if “new learning occur[red] as a result of this connectedness and interactivity, it emerge[d] from the network as a whole, rather than being transmitted or distributed by one or a few more powerful members”?
I honestly am having a hard time figuring out where/how to look for knowledge that wasn’t present in any of the individual communications, but emerges from the totality of them. But I think part of the problem here is that I don’t understand enough about Downes’ view of connectivism and connectivist knowledge. I knew I should take a closer look at connectivism before trying to tackle the question of evaluating cMOOCs! Guess I’ll have to come back to this after doing a post or two on Downes’ view of connectivism & connective knowledge.
Conclusion
So clearly I have a long way to go to understand exactly what Downes is suggesting and why, before I can even decide if this would be a good framework for evaluating a cMOOC.
In a later post I will look at two research papers that look at cMOOCs through the lens of Downes’ four process conditions, to see how they have interpreted and used these.
I welcome comments on anything I’ve said here–anything I’ve gotten wrong, or any suggestions on what I’m still confused about?
As noted in an earlier post, I have submitted some proposals for conference presentations on researching the effectiveness of connectivist MOOCs, or cMOOCs (see another one of my earlier posts for what a cMOOC is). I am using this post (and one or two later ones) to try to work through how one might go about doing so, and the problems I’ve considered only in a somewhat general way previously. I need to think things through by writing, so why not do that in the open?
I had wanted to think more carefully about connectivism before moving to some research questions about connectivist MOOCs, but for various reasons I need to get something worked out about possible research questions as soon as I can, so I’ll return to looking at connectivism in later posts.
Photo Credit: Detlef Schobert via Compfight CC-BY-ND
The general topic I’m interested in (at the moment)
And I mean general. I want to know whether we can determine whether a cMOOC has been “effective” or “successful.” That’s so general as to mean almost nothing.
What might help is some specification of the purposes or goals of offering a particular cMOOC, so one could see if it has been effective in achieving those. This could be taken from any of a number of perspectives, such as:
The connectivist MOOC model as implemented so far by people such as Dave Cormier, Alec Couros, Stephen Downes and George Siemens encourages participants to set their own goals and purposes for participation, rather than determining what these are to be for all participants (see, e.g., McAuley, Stewart, Siemens, & Cormier, 2010 (pp. 4-5, 40); see The MOOC Guide for a history of cMOOC-type courses, and lists of more recent connectivist MOOCs here and here). As Stephen Downes puts it:
In the MOOCs we’ve offered, we have said very clearly that you (as a student) define what counts as success. There is no single metric, because people go into the course for many different purposes. That’s why we see many different levels of activity ….
Further, just what a cMOOC will be like, where it goes, what people talk about, depends largely on the participants–even though there are often pre-set topics and speakers in advance, the rest of what happens is mostly up to what is written, discussed, shared amongst the participants. The ETMOOC guide for participants emphasizes this:
What #etmooc eventually becomes, and what it will mean to you, will depend upon the ways in which you participate and the participation and activities of all of its members.
Thus, it’s hard to say in advance what participants might get out of a particular cMOOC, in part because it’s impossible to say in advance what the course will actually be like (beyond the scheduled presentations, which are only one of many parts of a cMOOC).
Some possible directions for research questions
Developing connections with other people
I at first thought that perhaps one could say cMOOCs should allow participants to, at the very least, develop a set of connections with other people that are used for sharing advice, information, comments on each others’ work, for collaborating, and more. As discussed in my blog post on George Siemens’ writings on connectivism, what may be most important to a course that is run on connectivist principles is not the content that is provided, but the fostering of connections and skills for developing new ones and maintaining those one has, for the sake of being able to learn continually into the future.
And even though I understand what Downes and others say about participants in cMOOCs determining their own goals and deciding for themselves whether the course has been a success, cMOOCs have been and continue to be designed in certain ways for certain reasons, at least some of which most likely has to do with what participants may get out of the courses. Some of those who have been involved in designing cMOOCs have emphasized the importance of forming connections between people, ideas and information.
Stephen Downes talks about this in “Creating the Connectivist Course” when he says that he and George Siemens tried to make the “Connectivism and Connective Knowledge” course in 2008 “as much like a network as possible.” In this video on how to succeed in a MOOC, Dave Cormier emphasizes the value of connecting with others in the course through commenting on their blog posts, participating in discussion fora, and other ways. The connections made in this way are, Cormier says, “what the course is all about.” Now, of course, Cormier states at the beginning and end of the video that MOOCs are open to different ways of success and this is just “his” way, but the tone of the video suggests that it would be useful for others as well. Cormier says something similar in this video on knowledge in a MOOC: participants in a MOOC “are [ideally?] going to come out with a knowledge network, a network of people and ideas that’s going to carry long past the end of [the] course date.”
So it made sense to me at first to consider asking about the effectiveness or success of a cMOOC through looking at whether and how participants made connections with each other, and especially whether those continue beyond the end of the course. But again, there are some complications, besides the important questions of just how to define “connections” so as to decide what data to gather, and then the technical issues regarding how to get that data.
Would we want to say that the course succeeded more if more people made connections to others, rather than less? Or how about the question of how many people each participant should ideally connect with–I don’t think more is necessarily better, but where do we draw the line to say that x number of people made y number of connections with others, so the course has been a success?
This is getting pedantic, but I’m trying to express the point that when you really dig into this kind of question and try to design a research project, you would have to address this kind of question, and it’s kind of ridiculous. It’s ridiculous because there are so many different ways that connecting with other people could be valuable for a person, and for one person, having made one connection ends up being much more valuable than for another who has made 50. So much depends on the nature and context of those connections, and those are going to be highly individual and likely impossible to specify in any general way.
Further, what if some participants are happy to watch a few presentations and read blogs and lurk in twitter chats but don’t participate and therefore don’t “connect” in a deeper sense (than just reading and listening to others’ work and words). Should we say that if there are a lot of such persons in a cMOOC, the course has not been successful? I don’t think so, if we’re really sticking to the idea that participants can be engaged in the course to the degree and for the reasons they wish.
One possibility would be to ask participants to reflect on the connections they’ve made and whether/why/how they are valuable. One might be able to get some kind of useful qualitative data out of this, and maybe even find some patterns to what allows for valuable connections. In other words, rather than decide in advance what sorts of connections, and how many, are required for a successful cMOOC, one could just gather data about what connections were made and why/how people found them valuable. If done over lots of cMOOCs, one might be able to devise some sort of general idea of what makes for valuable connections in cMOOCs.
But would it be possible to say, on the basis of such data, whether a particular cMOOC has been successful? If many people made some connections they found valuable, would that be more successful than if only a few did? Again, this leads to the problems noted above–it runs up against the point that in cMOOCs participants are free to act and participate how they wish, and if they wish not to make connections, that doesn’t necessarily have to mean the course hasn’t been “successful” for them.
Looking at participation rates
photo credit: danielmoyle via photopin CC-BY
One might consider looking at participation rates in a cMOOC, given that much of such a course involves discussions and sharing of resources amongst participants (rather than transferral of knowledge mainly from one or a few experts to participants). As this video by Dave Cormier demonstrates so well, cMOOCs are distributed on the web rather than taking place in one central “space” (though there may be a central hub where people go for easy access to such distributed information and discussions, such as a blog hub), and this means that a large part of the course is happening on people’s blogs, on Twitter, on lists of shared links, and elsewhere. So it would seem reasonable to consider the degree to which participants engage in discussions through these means. How many people are active in the sense of writing blog posts, commenting on others’ blog posts, participating in Twitter chats and posting things to the course Twitter hashtag, participating in discussion forums (if there are any; there were none in ETMOOC) or in social media spaces like Google+, etc?
This makes sense given the nature of cMOOCs, since if there were no participation in these ways then there would be little left of the course but a set of presentations by experts that could be downloaded and watched. Perhaps one could say that even if we can’t decide exactly how much participation (or connection, for that matter) is needed for “success,” an increase in participation (or connection) over time might indicate some degree of success.
But again, we run up against the emphasis on participants being encouraged to participate only when, where and how they wish, meaning that it’s hard to justify saying that a cMOOC with greater participation amongst a larger number of people was somehow more effective than one in which fewer people participated. Or that a cMOOC in which participation and connections increased over time was more successful than one in which these stayed the same or decreased (especially since the evidence I’ve seen so far suggests that a drop off in participation over time may be common).
Determining your own purposes for participating in a cMOOC and judging whether you’ve reached them
Another option could be to ask participants who agree to be part of the research project to state early on what their goals for participating in the cMOOC are, and then towards end, and even in the middle, perhaps, ask them to reflect on whether they’re meeting/have met them.
Sounds reasonable, but then there are those people–like me taking ETMOOC–who don’t have a clear set of goals for taking an open online course. I honestly didn’t know exactly what I was getting into, nor what I wanted to get out of it because I didn’t understand what would happen in it. And as noted above, even though there may be some predetermined topics and presentations, what you end up focusing on/writing about/commenting on in discussion forums or others’ blogs/Tweeting about develops over time, as the course progresses. So some people may recognize this and be open to whatever transpires, not having any clear goals in advance or even partway through.
For those who do set out some goals for themselves at the beginning, it could easily be the case that many don’t end up fulfilling those particular goals by the end, but going in a different direction than what they could have envisioned at the beginning. In fact, one might even argue that that would be ideal–that people end up going into very different directions than they could have imagined to begin with might mean that the course was transformative for them in some way.
Thus, again, it’s difficult to see just how to make an argument about the effectiveness of a cMOOC by asking participants to set their goals out in advance and reflect on whether or not they’ve met them. Perhaps we could leave this open to people not having any goals but being able to reflect later on what they’ve gotten out of the course, and open to those who end up not meeting their original goals but go off in other valuable directions.
This would mean gathering qualitative data from things such as surveys, interviews or focus groups. I think it would be good to ask people to reflect on this partway through the course, at the end of the course, and again a few months or even a year later. Sometimes what people “get out of” a course doesn’t really crystallize for them until long after it’s finished.
Conclusions so far
It seems to me that there is a tension between the desire to have a course built in large part on the participation of individuals involved, and the desire to let them choose their level and type of participation. In some senses, cMOOCs appear to promote greater participation and connections amongst those involved, while also backing away from this at the same time. I understand the latter, and I appreciate it myself–that was one of the things that made ETMOOC so valuable for me. I was encouraged to choose what to focus on, what to write about, which conversations to participate in, based on what I found most important for my purposes (and based on how much time I had!). There are potential downsides to this, though, in that participants may not move far beyond their current beliefs, values and interests if they just look at what they find important based on those. But overall, I see the point and the value. I expect there are some good arguments in the educational literature for this sort of strategy that I’m not aware of.
Still, this is in tension, to some degree, with the emphasis on connecting and participating in cMOOCs. Perhaps the idea is that it would be good for people to do some connecting and participating, but in their own ways and on their own time, and if they choose not to we shouldn’t say they are not doing the course “correctly.” It might nevertheless be possible/permissible to suggest that, given the other side of this “tension,” looking at participation or connection rates could be considered as part of looking at the success of a cMOOC? Honestly, I’m torn here.
[Update June 7, 2013] I just came across this post by George Siemens, in which he doubts the value of lurking, at least in a personal learning network (PLN). There are likely differences of opinion amongst cMOOC proponents and those who offer them, on the value of letting learners decide exactly how much to participate.
It is, of course, possible that the whole approach I’m taking is misguided, namely trying to determine how one measure whether a cMOOC has been successful or not. I’m open to that possibility, but haven’t given up yet–not until I explore other avenues.
I had one other section to this post, but as it is already quite long, I moved that section to a new post, in which I discuss a suggestion by Stephen Downes as to how to evaluate the “success” of MOOCs. In that and/or perhaps another post I will also discuss some of the published literature so far on cMOOCs, and what the research questions and methods were in those studies.
Please comment/question/criticize as you see fit. As you can tell, I’m in early stages here and am happy for any help I can get.
I have submitted a proposal to two different conferences, for a session in which we would discuss the possibility and methods of researching the effectiveness of cMOOCs. One of those conferences is still considering the proposal, but as soon as decisions are made there I’ll post the proposal itself here on my blog. The proposal was accepted for a poster presentation at one conference, but I’m still waiting to hear from the other.
If the session gets accepted, I’ll need to give some background to cMOOCs in the way of talking about connectivism. And I want to dig my way through connectivist ideas anyway, so I’m going to do so here on the blog. That’s the way I think through things best–writing about them (or teaching, but I’m on sabbatical at the moment and not teaching).
I have read a number of articles and blog posts by George Siemens on connectivism, and have bookmarked quite a few others by Stephen Downes. Here I’ll discuss Siemens’ arguments, at least those I’ve found so far. I will not address the question of whether this is really a “new” learning theory, or whether it’s a learning theory at all, which are some issues that have been discussed in the research literature. I’m also not going to comment on the relationships between connectivism and constructivism, behaviourism, and cognitivism, as I have a woeful lack of knowledge of such theories. I’m just going to try to figure out some of (not all of) the basic ideas/arguments in what Siemens has written, and give my comments.
Context for the view
Siemens argues that connectivism makes sense for a context in which people have relatively access to a very large amount of information (through, e.g., the world wide web–not saying that everyone does have such access, but for those who do, Siemens is claiming, connectivism makes sense), can use technology to store that information rather than needing to have it in their own heads, and in which what counts as “knowledge” changes rapidly such that it becomes obsolete relatively quickly compared to past centuries and even decades (Siemens, 2005a). He claims we need a new learning theory, a new way of understanding how learning and knowledge work, within this sort of context.
Learning as a process of forming connections
To me, this is one of the fundamental ideas in connectivism, and the one I’m most interested in. I want to pick apart some of what Siemens says about it, in order to understand it better.
Learning is a process of connecting specialized nodes or information sources. (Siemens, 2005a)
I perceive learning as a network formation process. (Siemens, 2006b)
To really get at what is going on here, one would probably need to know more about learning theory than I do (as in, something about it, which I don’t). But the general idea is that when one learns something, what happens is that one makes connections between…what? Nodes. What counts as a “node”?
Siemens explains that networks have both nodes and connections, and “a node is any element that can be connected to any other element. A connection is any type of link between nodes” (Siemens, 2005b). He notes in (Siemens 2005a) that nodes can be, e.g., “fields, ideas, communities,” among other things. He also speaks of people as nodes. In this presentation posted by the Universitat Oberta de Catalunya (starting at around 7 minutes), Siemens describes teaching a course as a process of directing the formations of connections for students–when we given them particular course content, particular texts, particular theories to study and discuss, we are guiding how they form connections. The scope of “nodes” is very wide:
Virtually any element that we can scrutinize or experience can become node. Thoughts, feelings, interactions with others, and new data and information can be seen as nodes. (Siemens, 2005b)
Thus, connections can be made between nodes as persons, as ideas, as sets of data, as texts (or other media, such as videos), as groups, and more. So if I learn something, I make a connection (in my mind?) between myself and, say, a text, and between ideas I already have and those I’m getting from the text. I think that’s right, but I’m not absolutely certain, especially about the location of the connections–can some connections be located in thoughts, others outside? Siemens is clear that “[l]earning may reside in non-human appliances” (Siemens, 2005a), so clearly he thinks the connections don’t have to be only internal. But can they be internal at all? I’ll return to this question below.
First, briefly: How can learning reside in non-human appliances? If learning is a process of making connections, then appliances such as computers, and the software that runs on them, as well as the whatever that nebulous thing is that is sometimes called the “web,” could be considered as facilitating the making of connections. I make connections between myself and other persons, between my ideas and theirs, between my ideas and new information, quite often these days through the medium of these non-human entities. I suppose it is in that sense that learning, as a process of connection formation, can “reside” in non-human appliances. In a blog post entitled “What is the Unique Idea in Connectivism?”, Siemens explains the role of technology a bit further:
… technology plays a key role of 1) cognitive grunt work in creating and displaying patterns, 2) extending and enhancing our cognitive ability, 3) holding information in ready access form (for example, search engines, semantic structures, etc). (Siemens, 2006d)
Under (1), technological appliances like computers and software can create links and patterns, but also (2) extend our cognitive ability, which to me means that we can think through and understand many things more quickly and easily when we can quickly see and read and watch a number of resources about them. (3) is related to this too–the information is stored and readily accessible (well, sometimes readily…sometimes it’s quite hard to find) so that we don’t have to store it in our own, individual minds. The latter is true of textual technology like books, too. So if learning is a process of forming connections, then non-human entities can be and often are an important part of that process.
One might want to object that the appliances merely make possible these connections, that the connections themselves occur, somehow, mostly internally to individuals. One might think of the connection between some ideas I have already and some new ones I am introduced to in this way–the connection between these, however that might be characterized (and that’s a big question), seems to be localized in my own mind.
But what about the “connection” between myself and someone I communicate with entirely through the internet and the applications that allow me to do so? In what does it consist? Is it an abstraction in my mind? A feeling I have that I am linked to someone? Perhaps it makes more sense to think of this connection in terms of thoughts, feelings, plus tangible evidence of the connections in the form of emails, posts to social networks like Twitter, Google+, Facebook, video chats on Google Hangout or Skype, work that is collaboratively produced, and more. Some of these connections can be traced and visualized, such as connections on twitter being tracked and visualized through Martin Hawksey’s Twitter Archiving Google Spreadsheet (TAGS). Here’s a spreadsheet I made through TAGS for the #ds106zone Twitter hashtag, for May 23-31, 2013. And here’s the visualization of the connections on that hashtag for those seven days.
Thus, connections could, I think, be both in an individual’s mind in some way (however one might understand connections between ideas), and also located outside an individual as well.
Learning and knowledge
In reading some of Siemens’ articles and blog posts, I found myself getting confused as to the exact meanings of these tw0 terms, so I want to explore them further here. Learning is discussed briefly above as the process of making connections between nodes. What about knowledge?
Knowledge is defined as a particular pattern of relationships and learning is defined as the creation of new connections and patterns as well as the ability to maneuver around existing networks/patterns.
Our knowledge resides in the connections we form – where to other people or to information sources such as databases. (Siemens, 2006d; emphasis mine)
Siemens here suggests that while learning is a process of creating connections and patterns (as well as the ability to move around existing ones…though frankly I’m not quite sure what that means), knowledge is a particular pattern itself. In “Connectivism: Learning theory or pastime of the self-amused?” he states that knowledge “resides in a distributed manner across a network,” rather than being only in the mind of an individual (Siemens, 2006a, p. 10).
This “externalization of our knowledge,” he states in the same article, “is increasingly utilized as a means of coping with information overload” (p. 11):
Most learning needs today are becoming too complex to be addressed in “our heads”. We need to rely on a network of people (and increasingly, technology) to store, access, and retrieve knowledge and motivate its use. (Siemens, 2006c)
We have access to and can use so much information that we must externalize it through various means, such as storing it in handwritten notes, printed papers, or digital works such as texts, images, videos.
But what does it mean, exactly, to say that knowledge is a certain “pattern” of connections? The best way I can make sense of this for myself is with an example. Let’s say I know how to make an animated gif. We’ll ignore for the moment how it is I know that I know this, if I’m just trying to figure out the theory that would explain knowledge in the first place. How did I learn how to do this? I connected to a course called “ds106” (Digital Storytelling 106), which connected me to the instructor (Jim Groom), who connected me (through Twitter) to two tutorials on how to do it–a wiki page and a video. What/where is my “knowledge” of how to do this? Partly in my head, but partly not because I can’t (yet) remember each step. So partly it’s in the tutorials and the links I have to those on my computer, and partly in my link to the instructor whom I could ask questions of, and partly in my links to the other participants in the course who could help me as well. I can see, then, why one might say that the “knowledge” is not just what’s in my head, but also in some way “in” these connections. Of course, I could get to the point where I remember how to make an animated gif and so don’t need to access those connections for the basic procedure, but I would need to access the people and/or a web search if I wanted to do anything more advanced with gifs (which I did with the one linked above).
But what I’d like to see is some clearer and more detailed arguments on what counts as “knowledge” to justify why I should think of these connections I have to information on the web and to other people as part of my knowledge set. I am not an epistemologist, and haven’t studied epistemology since grad school, so I can’t go very far in criticizing this view from a philosophical perspective. I would, nevertheless, like to see a more fully worked-out argument for this view of knowledge.
In the connectivist view according to Siemens, then, it seems that learning is the process of creating patterns through developing connections, and knowledge is a resulting pattern within a network. Both, I think, can have both internal and external elements (knowledge can be a pattern of connections as abstract ideas and their logical links, e.g., as well as a connection stored in a computer text or video file).
But I get confused reading some of Siemens’ texts, because at times these words are defined slightly differently. For example, in Siemens (2006a), learning is stated to be the network itself: “The learning is the network” (p. 16), whereas I was thinking of learning as the process of making connections and then knowledge residing in the network thus created. He says the same thing in a blog post: “The network itself becomes the learning” (Siemens, 2006c). Perhaps this ambiguity is just the result of working this view out over a series of writings, which would be completely understandable. Blog posts, after all, may be treated by their authors as drafts of ideas, working through one’s views over time (I certainly view mine that way).
“The pipe is more important than the content within the pipe” (Siemens, 2005a)
This claim reflects the idea that since what counts as our “knowledge” is not only in our own heads but also outside of us, in a pattern of connections that is located partly in our own minds (and neurons!), partly in connections stored on computers, on paper, or technology (as discussed above), and partly in the links we have to other people, then what is most important is not what we have in our minds at any given moment, but the nature of these connections. When we have a problem we need to solve, for example, we don’t have to turn to the knowledge stored in our brains, but can turn to a web search, to people we’re connected with, to a course, or other sources to get the information/skills we need. “As knowledge continues to grow and evolve, access to what is needed is more important than what the learner currently possesses,” so “[n]urturing and maintaining connections is needed to facilitate continual learning” (Siemens, 2005a).
This, of course, has implications for teaching and learning: if we were to follow the connectivist view as teachers, we would not emphasize providing content to students. As many of us have already realized, at least some of the content we could provide is readily available to students on the web (This depends on the course, of course; I would say that my own interpretations of philosophical texts are not readily available, even though students could find others’ interpretations on the web pretty easily. But then again, if I post my interpretations as lecture notes on the web, then it would be available to them already.) Instead, the instructor could spend time with the class discussing, criticizing, asking and answering questions, etc.
And we could also help them with their “connectivist” skills–for lack of a better term (my term). We could help them with finding and evaluating information and information sources, for example, and with forming a network of people that can help them (and that they can help) in regards to a particular topic/field. Siemens (2008) provides a summary of various activities and roles for “connectivist” educators.
My current thoughts
Besides some slipperiness in terminology, the basic idea here makes sense. We could think of learning as a process of making connections, and knowledge as the patterns of connections thus made (if, that is, I’ve got the view right, which I might not of course). And in a context in which the internet makes information fairly easy to access (recognizing the problems with search engines filtering results in various ways) and connections to other people fairly easy to make (recognizing that people are most likely going to connect to others who are connected to who they already know, and those they tend to agree with most), I can see that the ability to make and access connections would be more important than what one “knows” in the sense of having information stored in one’s brain. One could also think of all learning as a matter of connecting things, whether it be making a connection to a book, a web page, a person, a video, etc., one might say that I am learning through making connections. I am also learning through adding the new information and skills I get thereby to my existing set, and making connections in that way as well.
I’m not convinced that this is the best way to think of learning and knowledge, at least not yet. The main problem is that I know nothing of learning theory, so I don’t know the other options. Another problem is, as noted above, I don’t think Siemens has a clearly worked-out, detailed epistemological view in the articles and blog posts I’ve read (as a philosopher I want many more specific and clear arguments supporting this view of knowledge). So while I think it makes some sense, I’m not convinced at the moment that I should accept Siemens’ view of connectivist understandings of learning and knowledge.
I think, however, that Stephen Downes has more arguments about connectivist epistemology in his writings, so that is who I’ll turn to next, in an upcoming post.
Your thoughts
Have I done justice to Siemen’s arguments about connectivism? What do you think of them? Please let us know in the comments!
Works cited
Siemens, G. (2005a). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning, 1(2). Retrieved from http://itdl.org/journal/jan_05/article01.htm
Siemens, G. (2005b). Connectivism: Learning as network-creation. Retrieved from http://www.elearnspace.org/Articles/networks.htm
Siemens, G. (2006a). Connectivism: Learning theory or pastime for the self-amused? Retrieved from http://www.elearnspace.org/Articles/connectivism_self-amused.htm
Siemens, G. (2006b, April 6). Learning, assessment, outcomes, ecologies. [Web log post]. Retrieved from http://www.connectivism.ca/?p=57
Siemens, G. (2006c, June 21). Constructivism vs. connectivism [Web log post]. Retrieved from http://www.connectivism.ca/?p=65
Siemens, G. (2006d, Aug. 6). What is the unique idea in connectivism? [Web log post] Retrieved from http://www.connectivism.ca/?p=116
Siemens, G. (2008). Learning and knowing in networks: Changing roles for educators and designers. IT Forum. Retrieved from http://itforum.coe.uga.edu/Paper105/Siemens.pdf
