Mid-Term SEoT pilot
ETEC565A Winter Term Two Mid-Term Formative Student Evaluation of Teaching Pilot
(Click here to download this page as a pdf)
The following data are from two sections of ETEC565A this term, where I piloted a mid-term formative student evaluation of teaching (SEoT) survey. I used WebCT Vista’s assessment tool to create an anonymized survey. In terms of response rate, 19 students completed the entire (brief) survey out of 34 registered in the course: six other students completed it at least partially. The survey was open for one week, in order to give me time to analyze the data and integrate any changes into the course contents and how I deliver the course.
Quantitative data
Students were asked seven Likert scaled questions about specific aspects of the course. Overall, there seems to be one student who is unhappy with the course and me; regardless, in most measures (except one) students view their experiences in the course so far as positive. The one exception was the question about the live session I delivered via Wimba Classroom: 7/18 viewed this session as positive; another 6/18 did not participate live but reviewed the archive of the session. Two more viewed the session negatively, and 3/18 were unsure. On one measure there was unanimity: everyone thinks my presence in the course has been “just the right amount”. Since I work very hard to get this aspect of online teaching right, instructor presence, this pleases me.
Below are data tables and pie charts of the responses to each of the questions that used Likert scales. These data are aggregated across both sections. For the “presence” question there is no chart, since the distribution of responses is zero.
Q1: The materials in this course site are well organized.
| Strongly Disagree | 3 |
| Disagree | 0 |
| Not sure | 1 |
| Agree | 3 |
| Strongly Agree | 12 |
|
# |
19 |
Q2: Select the response that best completes this sentence: The materials in this course site provide ______ detailed information
| Too much | 0 |
| Not enough | 2 |
| An adequate | 11 |
| Exactly the right amount of | 6 |
|
# |
19 |
Q3: The weekly discussions are integral to my learning in this course.
| Strongly Disagree | 1 |
| Disagree | 0 |
| Not sure | 5 |
| Agree | 9 |
| Strongly Agree | 10 |
|
# |
25 |
Q4: Exploring the eLearning toolkit in a self-directed manner is integral to my learning in this course.
| Strongly Disagree | 0 |
| Disagree | 0 |
| Not sure | 3 |
| Agree | 9 |
| Strongly Agree | 4 |
|
# |
16 |
Q5: Being part of a learning community has been a positive aspect of my experience in this course.
| Strongly Disagree | 1 |
| Disagree | 2 |
| Not sure | 1 |
| Agree | 7 |
| Strongly Agree | 9 |
|
# |
20 |
Q6: The instructor’s presence in this course has been:
| Minimal | 0 |
| A bit too little | 0 |
| Just the right amount | 19 |
| A bit too much | 0 |
| Way too much | 0 |
|
# |
19 |
Q7: The Wimba Classroom “Moodle Live” was very helpful.
| Strongly disagree | 1 |
| Disagree | 1 |
| Neutral/Not sure | 3 |
| Agree | 5 |
| Strongly agree | 2 |
| I did not participate in the session, but I viewed the archive | 6 |
| I did not participate in the session, nor did I view the archive | 0 |
|
# |
18 |
Qualitative data
For the qualitative data the response rate was 12/19 who completed the survey; 12/34 of all students enrolled, a bit more than one-third of the students across both sections. Of the 19 “responses” seven were blank or (the equivalent of) “no comment.” The students original unedited responses are Appendix A of this document; below I pull out a few concepts and respond to them. I will also post these responses in each course site, along with this report.
To students
In terms of the nice things several of you said, I appreciate—deeply—the positive feedback. It makes a huge difference to me. So thanks very much. But I will dwell on the issues that folks surfaced that need responses.
Elementary Education example: OK I’m simply embarrassed about this—and shocked no one’s pointed it out in the last 3 years. I’ll sort this out before the course is offered again.
WebCT Vista the dinosaur: I come to bury Vista, not to praise it. More seriously, this is the last time ETEC565A will be delivered via WebCT Vista. I was given the option to include this course in the current Blackboard Learn pilot, but given the range of technologies we need to manage already in here—delivering the course and for complete the learning activities—my instinct said “too much”. From what I’m hearing, this was absolutely the right decision. But the September offerings will be in Blackboard: guess I know what I will be doing this summer…
Resource for new UBC (graduate, online) students: I know that folks admitted to the MET are given access to some resources; I also know that unclassified students—or students taking this as their own MET course as part of other programs in which they are registered—might not get this (or any) such materials. For folks “going it alone” the onus is on you to find such resources—whether that’s fair or not, I’m not sure.
These all come from the http://wiki.ubc.ca/DE_Learning_Resources Distance Education Learning Resources page on the UBC Wiki.:
- http://learningcommons.ubc.ca/get-started/learning-environments/online-learning/Online Learning: Students’ Perspectives
- http://learningcommons.ubc.ca/get-started/learning-environments/online-learning/8-things-to-get-a-handle-on-online-learner-competencies/8 Things To Get a Handle On: Online Learners’ Competencies
- http://learningcommons.ubc.ca/get-started/learning-environments/online-learning/communicating-online/ Communicating Online
- http://learningcommons.ubc.ca/get-started/study-toolkits/online-groupwork/managing-conflict/ Managing Conflict
The Faculty of Graduate Studies (FOGS) offers:
- http://www.grad.ubc.ca/current-students/gps-graduate-pathways-success Graduate Pathways to Success workshops
- http://www.grad.ubc.ca/current-students/gps-graduate-pathways-success/resources-achieving-success Resources for (graduate) Student Success
These are just a sample. There is an expectation that graduate students will be self-directed…but no one expects folks to struggle on their own.
Proposal assignment assessment: the requirement for which folks got credit for included relevant course readings was “Integration of learning activities from relevant ETEC 565A activities”. Readings met the “learning activities” requirement, but so did the LMS rubric task, any of the discussions we had, or your own early explorations with the toolkit or with Moodle itself. The instructions also says you need to cite relevant literature: it does not quantify how many readings are required. I suggest anyone who believes that citing a single source for an assignment in a graduate course is acceptable should revisit that belief: it will not serve you well.
Regarding rubrics: In a graduate course I do not—purposefully—released detailed assessment rubrics in advance (though most folks could, with a bit of analytical thinking, glean somewhat how these are calculated). My expectation is you work to meet all the requirements, not “cherry pick” some based on their (relative) value. I do not accept that practices common in the primary, middle years or secondary space regarding assessment belong in higher education.
Moodle live Wimba session: a fair comment; in previous terms this session has been one or two weeks earlier. Doing it any earlier would conflict with the LMS rubric activity (weeks three and four) and the proposal assignment (week five). Doing it before we have reviewed much of module two would be before you’ve gained any context on LMS in general from the course materials. Week six is the earliest it can be scheduled: this term is was week eight.
This term I had previous commitments that prevented offering it in weeks six or seven—ones made before this course was scheduled. Like all of you, I too have a “day job” that impacts how I timetable this course. But this was an unusual convergence of circumstances.
Elearning toolkit: the expectations about the toolkit are communicated in the Course Introduction module on the Course Components page. It is designed to be wholly self-directed: as students work through the various learning activities in the course a good place to start is the toolkit. Folks who have been working on their LMS course site requirements will find it particularly useful.
The activities in the toolkit are all geared towards persons with no knowledge or experience with the technology or issue explore. They’re not, however, a place to find obvious “answers” to meeting summatively assessed assignment requirements.
Toolkit could be more comprehensive? It could! Feel free to email some suggestions on topics, including which ones you would like to develop—seriously! I did not create most of the current ones, after all…
Learning HTML: No one is required to learn HTML coding for this course—learning a bit of HTML code is exceedingly valuable for any learning technologist (hence the formative HTML quiz opened in the course on day one). But “learning HTML” does not mean creating web pages by writing your own code—it means using an application (there are several free ones in the toolkit, but many folks choose to try Dreamweaver) to create the pages. This allows you to design pages in a manner similar to a word processing application like MS Word.
You are free to build your sites using Moodle’s internal WYSIWYG HTML page creator/editor. But you will not be able to garner maximum points on the LMS course site assignment: Moodle’s tool does not create polished, refined pages—merely functional ones. So if you create a site using Moodle only—and your site meets all of the requirements of the assignment exclusive of the esthetics component of your mark—you will still get a good (equivalent to B+/A-) mark. The choice is yours. But in the realm of digital learning—online or blended—esthetics matter a great deal: they’re often what makes a learner/user experience positive or negative. One way to look at this: the caliber of your web design work is analogous to the quality of writing in an essay or thesis.
Constructivism: It is more labour intensive to manage your own explorations with respect to technical skills development in this course. For others, with a strong technical background but little pedagogical experience or training, the instructional design aspects of this course are the challenge. Most folks fall somewhere between these two extremes—which makes it impossible to create a course that hits the perfect level for everyone.
From my experience teaching courses like this—since the late 1990s—my leading you through an ongoing series of “how to activities” will leave most of you bored or restless at least half the time. More importantly, by requiring you to self-assess your skill level, seek out resources, and develop your own skills development activities you should experience more reliable skill transfer after the course. If you’ve ever done a workshop that’s very structured, only to find it difficult to retain your skills weeks or months later, you’ll know what I mean.
Thus, you are responsible for your own learning. Embracing this will make it much more likely you will retain the skills you acquire here—especially if you’re creating a site that is relevant to your own practice. But that does not mean you should struggle alone: I am happy to help folks who want to work on skills development, so long as you are not expecting me to “give you the answer” to the assignment requirements. And as a member of this course and the MET’s learning community, there’s a lot of other people willing to help besides me. If you ask—and you’re willing to do the work!
Synthesis reflection
I was keen to pilot this mid-term SEoT survey; after reviewing the data early on I was much less enthusiastic. Despite many years of teaching experience there remains for me a bruising element of SEoT. And I retain the ability to hone in on the negatives. I had to put these away for a couple of weeks after my first review.
In reality, these are not bad evaluations at all. There are several that are useful, in a couple of different ways. There are those that allow me to adjust things a bit for the rest of this term and in future terms. There are also those that create a teachable moment where I can (re)message a few key things. All of which is good.
Having said that, I am not committed to doing this in the future. In ETEC565A there is often around weeks seven and nine an irritability among students. They see the end of term looming and if they have not been able to project manage their work this term it can be a dispiriting time. They have also just completed their first largely technical assignment, which is a steep learning curve for many. Many are also starting to work on the web design aspect of their final LMS course site assignment—and are finding this not as easy as perhaps anticipated. All of which makes for some unhappy campers.
My experience with more traditional seminar courses—online or face-to-face—has been quite different. While I endeavor for the overall level of work across any graduate courses to be comparable, the nature of the work is important. For a seminar, students wrestle with the complexity of concepts rather than the complexity of tasks that’s found in ETEC565A.
Appendix A: Raw Qualitative Data
Response 1
I guess my main feedback at this point regarding the course is firstly I love all the information, very challenging discussions and ideas. Second – I find the technical learning quite overwhelming for someone who is new to many of the technologies, and has never designed an online course before – I do not even know what questions to ask half the time. But that said – I have done a lot of Moodling around and am starting to feel like I have a clue – some of my peers have been most helpful and encouraging, and the online Moodle community. I am not sure what other supports would work at this point.
Response 2
No, thank you
Response 3
I have found some of the assignments confusing. The e-toolkit is helpful but I am unsure if I am to complete the activities. I also think that it is assumed we are very comfortable with all technology. I was not aware I had to know HTML to complete this course. I understand that exploration is a way of learning but when we are expected to make a Moodle site with no experience or help it makes for a lot of long frustrating nights.
Response 4
Response 5
Now midway through the course I am finding the toolkit very useful but at the beginning of the course I must confess much of it was overwhelming. Now that I have participated in the wimba live chat I wished I had had the opportunity in the first week or two of class to “chat” about the activities in the course and maybe go over some of the concepts like html and how it relates to the final project etc. In the first weeks I was trying to climatize myself to the course overall and the html thing was beyond me to be honest. I did not understand the importance and significance of “leveraging outside software design” as was mentioned in the wimba chat. This may have forced me to dig a little deeper into free programs like dream weaver but I was so overwhelmed I did not. Now I am kicking myself in the pants! Overall this has been a highly engaging, very dynamic course. I have learned so much that I am sure when I have a moment to stop and reflect on the learning that transpired over the past 13 weeks I will be in awe! (-:
Response 6
The case studies have been very helpful, but at this point there has not been one dealing with Elementary school issues.
Response 7
John is extremely supportive and receptive of questions and new ideas. Keep up the good work!
Response 8
Impossible to please everyone! So far I have really enjoyed the course… however, as much as I like the concept of constructivism, I do not always have time for it. Most of us are teachers and are here to learn because we want to and many of us are also working at the same time… I’m finding that I’m having to spend too much time trying to figure Moodle out for myself. This is not the only course I’m taking and I’m working… I think the toolkit could maybe be more comprehensive and/or the Moodle live session could have been WAY earlier… I had already figured out half of what was shown… and if we are required to (or it’s highly recommended) to use an html editing program such as Dreamweaver, then that needs to be clearly stated with options to choose from in the assignment outline. It seems as though if we use the basic ‘page’ option within Moodle itself, that we will lose marks…
Response 9
This practical, hands on course is at a different pace and more self directed than the other MET courses that I have taken. This has taken a bit to get use to it but will be fine in the end.
Response 10
The audio in Wimba did not work for me, so it is hard for me to judge it. This is one of the most practical, hands-on course in MET which allows us to apply theory to practice.
Response 1
Response 2
Response 3
My only complaint is that WebCT is a bit of a dinosaur––but I suppose this is a moot point considering that it is being replaced. Having said that, I wish we had something like this at our school. =)
Response 4
I have enjoyed the course so far but really struggled with starting to set up my Moodle page. I think having the Moodle live session earlier would have been helpful.
Response 5
This is my first course with UBC and in this program. I find it difficult to find educational resources, I wish there was a new student handbook that outlined what UBC has to offer for resources for online students. If there is one already, I would love access to it!
Response 6
Response 7
The rubric for the first ‘Proposal’ assignment did not reflect my expectations; a minimum of three references were required for full marks, but the assignment only stated that one was required. However it’s not a significant issue.
Response 8
Not at this time.
Response 9
I have really enjoyed the involvement of the instructor (John). His comments show that he is following discussion, providing challenging comments when appropriate, correcting perceptions and just being playful. I really enjoy the summaries he provides on the cases. He is always accessible.