Tag Archives: Arts One

Presentation on SoTL research re: peer feedback

In mid-November I gave a presentation at the SoTL Symposium in Banff, Alberta, Canada, sponsored by Mount Royal University.

It’s a little difficult to describe this complex research, so I’ll let my (long) abstract for the presentation tell at least part of the story.


750-word abstract

Title: Tracking a dose-response curve for peer feedback on writing

There is a good deal of research showing that peer feedback can contribute to improvements in student writing (Cho & MacArthur, 2010; Crossman & Kite, 2012). Though intuitively one might think that students would benefit most from receiving peer comments on their written work, several studies have shown that student writing benefits both from comments given as well as comments received–indeed, sometimes the former more than the latter (Li, Liu & Steckelberg, 2010; Cho & MacArthur, 2011).

There are, however, some gaps in the literature on the impact of peer feedback on improving student writing. First, most studies published on this topic consider the effect of peer feedback on revisions to a single essay, rather than on whether students use peer comments on one essay when writing another essay. Cho and MacArthur (2011) is an exception: the authors found that students who wrote reviews of writing samples by students in a past course produced better writing on a different topic than those who either only read those samples or who read something else. In addition, there is little research on what one might call a “dose-response” curve for the impact of peer feedback on student writing—how are the “doses” of peer feedback related to the “response” of improvement in writing? It could be that peer feedback is more effective in improving writing after a certain number of feedback sessions, and/or that there are diminishing returns after quite a few sessions.

To address these gaps in the literature, we designed a research study focusing on peer feedback in a first-year, writing intensive course at a large university in North America. In this course students write an essay every two weeks, and they meet every week for a full year in groups of four plus their professor to give comments on each others’ essays (the same group stays together for half or the full year, depending on the instructor). With between 20 and 22 such meetings per year, students get a heavy dose of peer feedback sessions, and this is a good opportunity to measure the dose-response curve mentioned above. We can also test the difference in the dose-response curve for the peer feedback groups that change halfway through the year versus those who remain the same over the year. Further, we can evaluate the degree to which students use comments given by others, as well as comments they give to others, on later essays.

While at times researchers try to gauge improvement in student work on the basis of peer feedback by looking at coarse evaluations of quality before and after peer feedback (e.g., Sullivan & Pratt, 1996; Braine, 2001), because many things besides peer feedback could go into improving the quality of student work, more specific links between what is said in peer feedback and changes in student work are preferable. Thus, we will compare each student’s later essays with comments given to them (and those they gave to others) on previous ones, to see if the comments are reflected in the later essays, using a process similar to that described in Hewett (2000).

During the 2013-2014 academic year we ran a pilot study with just one of those sections (sixteen students, out of whom thirteen agreed to participate), to refine our data collection and analysis methods. For the pilot program we collected ten essays from each of the students who agreed to participate, comments they received from their peers on those essays, as well as comments they gave to their peers. For each essay, students received comments from three other students plus the instructor. We will use the instructor comments to, first, see whether student comments begin to approach instructor comments over time, and to isolate those things that only students commented on (not the instructor) to see if students use those in their essays (or if they mainly focus on those things that the instructor said also).

In this session, the Principal Investigator will report on the results of this pilot study and what we have learned about dealing with such a large data set, whether we can see any patterns from this pilot group of thirteen students, and how we will design a larger study on the basis of these results.


 

It turned out that we were still in the process of coding all the data when I gave the presentation, so we don’t yet have full results. We have coded all the comments on all the essays (10 essays from 13 participants), but are still coding the essays themselves (had finished 10 essays each from 6 participants, so a total of 60 essays).

I’m not sure the slides themselves tell the whole story very clearly, but I’m happy to answer questions if anyone has any. I’m saving up writing a narrative about the results until we have the full results in (hopefully in a couple of months!).

We’re also putting in a grant proposal to run the study with a larger sample (didn’t get a grant last year we were trying to get…will try again this year).

Here are the slides!

Lecture on Hobbes’ Leviathan

On Monday, Nov. 10, 2014, I gave a lecture on Hobbes’ Leviathan for Arts One at UBC. There is a video recording, but we don’t post those until after students have submitted their essays (so no one is tempted to skip lecture!). I’ll link to that when it’s ready.

I wanted to share my presentation slides here because, as usual, I didn’t get to everything I wanted to say (it’s so hard to gauge exactly how much you can fit into a 2-hour lecture (or rather, 2 50-minute lectures!). I wanted to let students (and anyone else who is interested) get a chance to see the last few slides.

Or rather, I used Prezi for the first time with this lecture. I like it because it allows you to group your slides together in ways that can show how the argument is structured. Only mine is a bit messy–what does that say about my argument in the lecture, eh?

Here’s the link to the Prezi itself (wish they had an embed function!)
http://prezi.com/c9u71sd-iegp/?utm_campaign=share&utm_medium=copy

I only got to the last three slides on the bottom (the powers of the sovereign, the liberty of subjects, and what subjects can’t do). If you saw the lecture, skip past those to see what else I was trying to say, and why I was gesturing towards thinking that maybe the Hobbesian state wouldn’t be monstrous, and maybe our state shares some similarities with a Hobbesian one.

Or, if you don’t want to go through the whole Prezi just to get to the last few slides, here they are (they’ll make sense, hopefully, by themselves if you saw the lecture).

The last three are zoomed into the frontispiece as if we were going into the body of the commonwealth; thus the grey background!

Screen Shot 2014-11-12 at 10.57.15 AM

Screen Shot 2014-11-12 at 10.57.28 AM

Screen Shot 2014-11-12 at 10.57.43 AM

Screen Shot 2014-11-12 at 10.57.57 AM

Screen Shot 2014-11-12 at 10.58.06 AM

Screen Shot 2014-11-12 at 10.58.19 AM

Screen Shot 2014-11-12 at 10.58.29 AM

The Power of Space in the Classroom

Most of us know very well the importance of space in the classroom–how the room is set up can really change the dynamics of a class. For example, in a discussion course, I try to set up the room in as much of a circle as possible (which, given the configuration of some rooms, is sometimes impossible). Once I had a seminar-style class in a room where we simply could not put the tables and chairs into a circle, and had to leave them in rows (because there wasn’t enough room to put them in a circle). That was the worst term I’ve ever had for discussion.

A colleague of mine in the Arts One Program was even more innovative in her use of space than I’ve ever thought of being myself.

I have had the chance to view the classes of some of my colleagues in Arts One over the past few years. I wish I had more such chances to see others teach, since I always learn from what others are doing in their classes.

Arts One has two, 75-80 minute seminar-style discussion classes per week, with a maximum of 20 students, so most of the rooms we have allow for circular (actually rectangular) seating. There are tables arranged in a circle, with a big space in the middle of them. That works pretty well, since everyone can see everyone else.

Still, the professor usually sits at one of the “heads” of the table, on one of the shorter ends (we don’t have to do this, of course, but I’ve often seen it done). Subtly, then, we are still making ourselves the focal point by making sure most students can see us well (often students avoid sitting right next to the prof, and sit on the longer sides of the table instead).

This sort of setup is good for having books, paper and computers (if they’re allowed) out on the desk while engaging in discussion, but the tables with the big space in the middle cuts us off from one another in a sense, providing a pretty big distance from one another.

Continue reading

Oral and written peer feedback

This post is part of my ongoing efforts to develop a research project focusing on the Arts One program–a team-taught, interdisciplinary program for first-year students in the Faculty of Arts at the University of British Columbia. As noted in some earlier posts, one of the things that stands out about Arts One is what we call “tutorials,” which are weekly meetings of four students plus the professor in which all read and comment on each others’ essays (students write approximately one essay every two weeks). Thus, peer feedback on essays is an integral part of this course, occurring as a regular part of the course meeting time, every week.

In a recent survey of Arts One Alumni (see my post summarizing the results), students cited tutorials as one of the things that helped them improve their writing the most, and as one of the most important aspects of the program. In that earlier post I speculated on what might be so valuable about these tutorials, such as the frequency of providing and getting peer feedback (giving feedback every week, getting feedback on your own paper every two weeks), the fact that professors are there in the meetings with students to give their comments too and comment on the students’ comments, the fact that students revisit their work in an intensive way after it’s written, that they may feel pressure to make the work better before submitting it because they know they’ll have to present and defend it with their peers, etc. That last point is perhaps made even more important when you consider that the students get to know each other quite well, meeting every week for at least one term (the course is two terms, or one year long, but some of us switch students into different tutorial groups halfway through so they get the experience of reading other students’ papers too).

One thing I didn’t consider before, but am thinking about more now, is whether the fact that the feedback is done mostly, if not exclusively, orally and synchronously (and face-to-face) rather than through writing and asynchronously, makes a difference.

Continue reading

Results of a survey of Arts One alumni, and thoughts on research questions

 

In March and April of 2012, a research assistant (an Arts One alum) and I did a survey of alumni of the Arts One program (a team-taught, interdisciplinary, year-long program for first year students at UBC). Arts One has been in existence since 1967, yet very little research has been done on how it impacts students. I hope to do some research on that broad question in the coming years.

The purpose of the survey of alumni from earlier this year was to see how students themselves thought Arts One impacted them, in two ways: how it impacted their work in other courses, and how it impacted them beyond their work in other courses. Just for fun, and to see if we could isolate that which really sets the Arts One program apart from other first year programs, we also asked them what they thought the most important aspect of Arts One was (a question I had taken from an earlier study on Arts One, by Cheryl Dumaresq (see https://circle.ubc.ca/handle/2429/3693 for a copy of Dumaresq’s survey of Arts One alumni in the 1990s). We surveyed students who had just finished Arts One that Spring, plus those who had taken Arts One their first year and were still at UBC, in their second, third, fourth years and beyond. 116 students answered our email request to fill in an online survey. The questions were all open-ended and subjected to descriptive qualitative analysis only.

The reason for doing this survey, beyond being interested generally, was to gain material for developing research questions to study further later. In other words, this was a pilot study to determine which areas of Arts One to subject to further research, and how/why. In what follows I begin thinking about some possible research questions arising from the data.

In some ways, the results of the survey were unsurprising, in the sense that I guessed before doing the survey much of what came out of it. But there were a couple of things I hadn’t thought of before, which was the point of doing the survey in an open-ended way (so students weren’t stuck with giving only multiple-choice answers to topics we thought of ourselves).


 

[What follows provides only a discussion of only a few of the prominent answers, or the ones I found most interesting. There were many, many different things said by students, with the result that most of those ended up being said only a few times, and thus are not recorded below unless I found them particularly interesting or surprising.]

Percentage of student alumni saying Arts One positively impacted their other coursework

Over 90% said Arts One impacted them in ways that positively affected their work in other courses. A few said there was no impact on their work in other courses, and a few said it impacted their work negatively (though some of those also said it had some positive impacts too).

How Arts One helps with other courses

  • Most of the respondents said it helped with writing (73% of respondents; 80% of those who said A1 had positive impact on other courses). This wasn’t surprising, as improving writing is one of the main emphases of Arts One. Students write 12 essays over the course of one year (approximately one every two weeks), and each of those essays is peer reviewed in a tutorial meeting of four students plus their professor. Every week students meet in these tutorials and discuss two essays, each for 25-30 minutes.
    • Of those, most said it was the tutorials that helped with their writing (48%), and next highest cited was the amount of writing (39%). Some of these may be the same people (they may have said both things).
    • What could be some reasons students think tutorials are so helpful for improving their writing? In tutorials we discuss particular issues with the papers being commented on, but also general advice about writing academic papers, about the writing process, etc. Some of this advice comes from the professors and some from the students. One thing that several students mentioned as being helpful is the chance to read and comment on others’ essays; some noted that this helped them think more critically about their own. One stated that giving comments to peers in front of the professor, and having the professor comment on those comments, was especially helpful. Students likely also feel some pressure to make their writing better because they have to present and defend it in front of peers and the professor. Revisiting work in an intensive way after it’s written may also contribute to improvement in writing.
    • Possible research areas:
      • It would be interesting to see if students’ self-perceptions match up with reality: does their writing really improve after completing the program? One could compare their improvement with gains in writing skill from other first-year writing courses (though that might be difficult, as several such courses at UBC have different foci than the writing in Arts One, emphasizing research whereas we do not). I could have a look at the literature on self-perceptions of skills (esp. writing skills) and objective measures of those, to see if generally self-perceptions tend to be accurate or not. I recall reading something about this somewhere, but I don’t have the details ready to hand.
      • I could focus on and expand upon the above questions about tutorials: what it is about tutorials leads students to cite them as particularly helpful for improving their writing skills? Is there something special about A1 tutorials, as opposed to peer review in other courses? E.g., I do peer review in other courses, and get no such comments about its value, even though the peer review is done in small groups during class time, as with the Arts One tutorials. Peer review does not constitute a separate class period in those other courses, though, like it does in Arts One, and I as the professor am not sitting there with the group the whole time they are doing the peer review. Is there something special about having the prof present the whole time? Is there something special about the amount of time spent on each essay? Is it that it is done every single week? Does the fact that it’s a scheduled part of the course, so especially emphasized, make a difference? Is it perhaps that students recognize that tutorials are supposed to help them with writing, so they think and say that they do? After all, one wouldn’t want to spend so much time doing tutorial work for nothing. I’m not sure how to even begin to answer any of those questions yet!

 

  • A significant number of students said Arts One contributed to their critical thinking skills (24% of respondents, 27% of those who said A1 had a positive impact on their other coursework). I expected this to show up in the survey, as one of the things we emphasize in Arts One is providing students with a lot of opportunity to develop their own responses and arguments to the texts and issues we discuss. The seminars are focused on discussion, and many of us try to get the students to lead that discussion as much as possible (with some variation for professor style, of course) and though we provide essay topics for the papers, they are purposefully open-ended so that students have a lot of room to argue for their own readings and emphasize what they find important.
    • However, the survey responses did not provide much clarity as to what aspects of the program especially helped with critical thinking, in the students’ views. Many of those who said A1 helped with critical thinking did not explain clearly how, and among those that did, numerous aspects were cited such that I couldn’t find a clear pattern for which things were most helpful.
    • possible research areas
      • Studying what critical thinking is, and how to promote it, is an entire field in its own right. There are numerous definitions of critical thinking, and quite a few different tests of it, and I would need to delve into that literature before even beginning to think further about this topic. It’s unclear what each student meant by terms like “critical thinking,” or “analysis skills” when they said them, as well.
      • One thing I am particularly interested in myself is perhaps captured by the term “critical thinking,” but perhaps not–it’s the ability and confidence to engage in independent thinking. I think what I mean by that term is that students feel they can sit down with a text and make sense of it on their own, or with a small group of peers, without needing to rely on finding out what others (experts) have said about it in order to find out what is the “right” way to read the text. We specifically and consciously downplay seeking outside research on the texts we read, so as to focus students’ attention on their own reading and analysis skills. I, personally, also emphasize that the idea in writing essays for Arts One is less a matter of getting something right about the text than it is about coming up with a thought-provoking, justified reading, an interpretation grounded in the text but that also goes beyond a surface level and can make the reader really think. I am hoping students take risks rather than provide just the safest or most clear-cut arguments possible. I would need to better clarify just what I am looking for here, and what terms that corresponds to in the literature.

 

  • The same number of students as said Arts One helped critical thinking said it helped their confidence in some way, such as confidence speaking in class, speaking to profs, confidence in their writing skills, or in their own ideas as valuable (24% of respondents, 27% of those who said positive impact).
    • I find this one especially interesting, perhaps because I hadn’t really thought about it before, and yet it is so important to students’ future coursework and life. My rough analysis of the survey data has not yet revealed clear patterns on how, exactly, students think Arts One helps with their confidence, which aspects do so and why. I plan to go back over the data and see if I can come up with something clearer on this, or if the answers are simply too thinly scattered.
    • possible research areas
      • Maybe something in the area of confidence would be a better place for me to focus my attention to try to capture what I was talking about at the end of the discussion on critical thinking. If students feel confident in their own ideas and their ability to express and defend them, they might be more likely to approach texts and discussions in the way I suggested.
      • Studying confidence has the added bonus of being easier to study than whether writing really improves or whether critical thinking really improves, because one would, presumably, rely on students’ own self-reports of confidence. Though I suppose that some behavioural data might contribute to showing increases or decreases in confidence as well.
      • There must be literature on levels of confidence and how this affects students’ coursework…I would need to look into that.


Percentage of student alumni saying Arts One positively impacted them beyond coursework

83% of students said that Arts One impacted them positively in ways beyond their work in other courses.  Most of those who did not assent to this simply said it did not impact them beyond coursework. Only two gave negative comments in response to this question, and only one of those was about the program itself.

How Arts One impacts students beyond their work in other courses

  • The most often-cited answer to this question from students was that Arts One provided a close-knit community that allowed them to develop friendships, get to know their colleagues and the professor, and feel comfortable in the classroom (38% of respondents, 46% of those who said Arts One impacted them in some positive way beyond their work in other courses). I guessed beforehand that this would be important, as it’s something else that we emphasize in Arts One. There are five professors who team-teach the course (two groups of five profs total, with separate themes, readings, and students); there are up to 100 students in the course and each professor is assigned to up to 20 of those. Each week there is a lecture given by one of the five professors to all 100 students, and two seminar discussions of 20 students with their professor. Then there are tutorials of four students from the group of 20 with their professor, and each student has one of these per week (each prof has five). The students in the seminar group of 20 get to know each other and their professor very well, with two discussions per week plus a tutorial every week. They often get together outside of class for study or social purposes, develop group Facebook pages, etc.
    • The seminars were cited as more important for developing a sense of community and friendships (25% of respondents, 30% of those who said A1 had positive impact beyond coursework) than tutorials (14% of respondents, 17% of those who said positive impact). I suppose this could be a little surprising, since the tutorials are only 4 students plus the professor, while the seminars are 20 students plus the professor. But the tutorials are somewhat stressful for numerous students, as it is where they have to present and defend their essays, listening to and responding to criticism from their peers and professor. They also have to learn to constructively comment on the essays of their peers, which can be difficult for many at first. Though by the end of the year the tutorials are often much more relaxed, I am not surprised that students don’t view tutorials as being as much of a space for developing a close sense of community and friendships as seminars, overall.
    • possible research areas
      • It might seem, and many students thought of it this way, that developing a sense of community and having friendships come out of Arts One is mainly a social effect that doesn’t have much to do with academic coursework. But I think that’s a mistake. I know I have read some things that point to the importance of having a close community and friendships in courses, and it would be good to revisit those to show that this could be an important ingredient to student success in Arts One. If students see the value of developing an academic and social community in their classes through their Arts One experience, it might encourage them to seek to develop those in other courses as well (as a few students mentioned in their answers to the survey). So looking at the literature on community and student success (broadly defined) would be a good place to start. Then I might be able to find best practices on developing a sense of community and see if they are implemented in Arts One. Or see if some aspects of Arts One are especially important to developing a sense of community that might be exported to other courses.


  •  A significant number of students pointed out that Arts One had improved their confidence in ways that extended beyond their work in university courses, such as confidence in public speaking, in their own views, and in writing (13% of respondents, 16% of those who said A1 had positive impact beyond coursework). This is less than the number who pointed to confidence in response to how Arts One had impacted their work in other courses, but presumably if it gave them confidence in the above ways for courses, most of that would transfer to their life beyond the university.
    • possible research areas: Same as above re: confidence, but it’s important to think about the value of confidence beyond how it can help students succeed in their university coursework.

 

The most important aspect of Arts One, according to student alumni

  • Most often cited as an answer to this question was having small classes (35% of respondents).
  • A close second were seminars and tutorials, which had about the same number of people citing them in answer to this question (31% of respondents said something about seminars in response to this question, and 32% said something about tutorials).
  • The next highest category was people who said something relating to the quality of the professors, a particular professor, or the lectures given by professors (26% of respondents).
  • Finally, a significant number of people said the most important thing was the ability to have close connections between students and professors (19%). This, of course, is closely connected to having small classes (though small classes are not required for it, they can help facilitate it). The 4-person tutorials are especially conducive to students being able to work closely with professors, and to gain confidence in speaking to them.
    • possible research areas
      • Many of the above are related–small classes (seminars and tutorials) and the ability to have a close connection between students and professors. Why is it that connecting with one’s professor is so important? What does it facilitate that being an anonymous face in a classroom where one never speaks to the professor does not? Does the fact that Arts One does not have any Teaching Assistants have a bearing on these issues, or would a close connection to Teaching Assistants yield similar results?

 

There is much to think about here. Obviously I’ve raised enough research areas to last a lifetime. I just need to pick the one I’d like to work on for the next few years…a daunting task!

 

Here is a more detailed report, without the possible research areas, and including the questions asked: A1AlumniSurvey2012-Report-Oct2012

 

Students reading/not reading the texts

(The last term, Jan-April 2012, was incredibly busy for me…no time to blog! But now I’m on a year’s sabbatical (July 2012-July 2013), and plan to do a good deal of reading and writing about teaching!)

Some common concerns: How can we engage in good classroom discussion of one or more texts and the issues and arguments they raise if a good number of students haven’t read the texts? How can we encourage students to read the texts before coming to class?

Clearly we need to motivate students so they want to read the texts before class. If the readings were intrinsically interesting to them and they had lots of time, it wouldn’t be a problem. Unfortunately, those two things are not always true (and with philosophical texts, the first is too often not true for many students). In addition, I have in the past found myself making up for the lack of preparedness by lecturing on the main ideas in the texts myself in class. Which, of course, just makes students even less likely to read the texts: if the texts are difficult, and the professor is just going to lecture on the main points anyway, why not spend one’s precious time in one’s very busy life doing something else? For many college students are very, very busy–probably too much so; but that’s an argument for another day.

One of the courses I teach is “Arts One,” a year-long, interdisciplinary, team-taught course for first year students only. In that course we read a book a week, approximately (sometimes one book over two weeks), and most of the class time is spent in discussion of the books. It is imperative that students have read the texts before coming to discussion class, and in that particular class, I have too often relied on the thought that the students who choose Arts One are already diligent and highly motivated, and would do the work on their own. But that isn’t always the case, and often it’s because they are overworked or the texts are difficult for them to understand on their own.

Things really become problematic when I ask students to do presentations either to the whole class or to small groups, in which they raise questions for discussion and facilitate that discussion. They are understandably disappointed when their hard work results in lackluster discussion because others haven’t finished reading the books.

Continue reading

Developing a SoTL project

I have just begun attending a year-long faculty certificate program on the scholarship of teaching and learning here at UBC. One of the main things this program is designed to do is to support faculty who wish to start engaging in the scholarship of teaching and learning (SoTL).

Just for quick reference, “scholarly teaching” refers generally to a practice where educators base their pedagogical practices on as much evidence as they can from relevant research, studies devoted to showing what is most effective for what sorts of desired outcomes, etc. SoTL, then, is doing this plus engaging in research oneself, and disseminating that somehow to colleagues (e.g., through conferences, publications, etc.).

Continue reading