Evidence of Learning in Astronomy

Posted on December 22, 2010 by Peter Newbury

Throughout the Sep-Dec, 2010 term, I worked with an astronomy instructor to create a more learner-centered classroom. As I described elsewhere, we spent just over one third of the instructional time on interactive activities: think-pair-share using clickers, lecture-tutorial worksheets, ranking tasks and a couple of predict-observe demonstrations. And it resulted in a learning gain of 0.42 on a particular assessment tool, the LSCI. That means the students learned 42% of the concepts they didn’t already know at the beginning of the term. That’s not bad — we’re pretty happy with it.

So, students can learn in a learner-centered classroom. But maybe they can learn in a more traditional classroom, too.

We don’t have LSCI data from previous years (note to self: think ahead! Collect standardized assessment data on classes before attempting any transformations!) To investigate if transforming the instruction class makes any difference, we re-used, word-for-word, a handful of questions from the same instructor’s 2008 Final Exam (pre-transformation) on this term’s Final Exam: 10 multiple-choice questions and 4 longer-answer questions. We made sure the questions assessed the concepts we covered in 2010 in sync with the learning goals.

I extracted students’ marks on these 14 questions from the 2010 exams (N=144). And from the old, 2008 exams (N=107), being sure to re-mark the longer-answer questions using the 2010 rubric. (Note to self #2: buy aspirin on the way home.)

What were we hoping for? Significant increase in student success in the transformed,  learner-centered course.

How I wish I could report that’s what we found. But I can’t. Because we didn’t. Here are the results:

Students scores on questions used on both the 2008 and 2010 Final Exams in the introductory astronomy course, ASTR 311. Error bars are standard error of the mean.

There is no significant difference in student success on the 10 multiple-choice questions. Their scores on the entire exams are also the same, though the exams are not identical, only about 1/4 of the 2008 exam is re-used in 2010. Nevertheless, these nearly identical Exam scores suggest the populations of students in 2008 and 2010 are about the same.  There are are differences in the 4 long-answer questions: the 2008 students did better than their 2010 counterparts.

Two things jumped out at me

  1. Why did they do so much better on the long-answer questions? I said we used the same marking rubric but we didn’t use the same markers. A team of teaching assistants marked the 2010 exams; I(re) marked the 2008 exams. The long-answer questions are work 10 marks, so a little more (or less) generosity in marking – half a mark here, half a mark there – could make a difference. I really need to get the same TAs to remark the 2008 exams. Yeah, like that’s gonna happen voluntarily. Hmm, unless there’s potential for a publication in the AER…
  2. Why, oh why, didn’t they do better this year? Even if we omit the suspicious long-answer marks and look only at the multiple-choice questions, there is no difference. Did we fail?

No, it’s not a failure. The instructor reduced her lecturing time by 35%. We asked the students to spend 35% of their time teaching themselves. And it did no harm. The instructor enjoyed this class much than in 2008. We had consistent 75% attendance (it was much lower by the end of the term in 2008) and students were engaged in material each and every class. I think that’s a success.

The next step in this experiment is to look for retention. There is evidence in physics (see Pollock & Chasteen, “Longer term impacts…” here) that students who engage in material and generate their own knowledge retain the material longer. With that in mind, I hope to re-test these 2010 students with LSCI in about 3 months, after they’ve had a term to forget everything. Or maybe not…

Posted in astro 101 | Tagged , , , , | 3 Comments

Leveraging Public Outreach

Posted on December 16, 2010 by Peter Newbury

Every winter for the last 7 years, my Department has put on a science show for the general public, following the tradition celebrated by physicist Michael Faraday (1791-1867).

Orion (Credit & Copyright: Matthew Spinelli)

This year’s theme was “The Physics of Light and Colour”. We did demos about lasers, solar power, optical illusions, twinkling stars, this cool Cubee device developed at UBC. For my 10 minutes, I talked about colour and temperature – how we usually portray cold as blue but to astronomers, blue = hot. I encouraged everyone to go home and look for the bright, coloured stars in the constellation Orion.  (I projected Star Walk from my iPad thru the theatre projector to show everyone where to look. Oh my! Fantastic!)

We had an amazing turn-out: nearly 350 people, at least half of them kids!

Before and after the show, I set up a four Galileoscopes on tripods outside the building. Sure, it was to attract people and tell them they were in the right place. But really, it was so I could get a bunch of kids looking through my telescopes. It was the middle of the day so we looked at signs, cars, people, a statue and a Caterpillar excavator. And they were all upside-down! What’s up with that, kids?

When it was over and I had a few quiet moments to take down the telescopes, I thought about the best parts of the afternoon, and came to a new realization about why I do “sidewalk astronomy”. From least to most important:

3. I love looking at stuff through my telescope. I can still remember the first time I saw Saturn’s rings. This is one of those defining moments for many budding astronomers, like watching Neil Armstrong take his one small step is the defining moment for almost every astronaut who’s ever talked about why they became an astronaut.

2. I love helping kids see stuff through my telescope. Typically, kids run up, grab the scope, jam their eyes near the eyepiece. Talk talk talk blab blab squeal. And then the image snaps into focus. “blab blab squ-. . . . . . oooooooh!” I love that moment! (Tip: if you’re aiming at something really bright like the Moon, often you see the image forming on the kid’s cheek and you can gently maneuver their head to get the bright image on their pupil.)

1. This is the new one for me. It was so rewarding talking to the kids’ parents. They’re already interested enough that they’ve brought their kids to your event. If you can get them enthusiastic and excited, they’ll carry on the conversation and discoveries at home. And they’ll be the ones who experience that magical moment when their kids see Saturn for the first time.

Unfortunately, it means I give up the golden moments with the kids — and that’s hard to do because it’s like a drug. But it leverages my enthusiasm and excitement. And knowledge: knowledge of astronomy and knowledge of how to teach astronomy.

Still, here I am, writing about how much I enjoyed talking with the parents. Maybe I’ve found a new drug…

Posted in communicating science, outreach | Tagged , , , | Leave a comment

But did they learn anything?

Posted on December 13, 2010 by Peter Newbury

The course transformations I work on through the Carl Wieman Science Education Initiative (CWSEI) in Physics and Astronomy at UBC are based on a 3-pillared approach:

  1. figure out what students should learn (by writing learning goals)
  2. teach those concepts with research-based instructional strategies
  3. assess if they learned 1. via 2.

Now that we’ve reached the end of the term, I’m working on Step 3. I’m mimicking the assessment described by Prather, Rudolf, Brissenden and Schlingman, “A national study assessing the teaching and learning of introductory astronomy. Part I. The effect of interactive instruction,” Am. J. Phys. 77(4), 320-330 (2009) [link to PDF].  They looked for a relationship between the normalized learning gain on a particular assessment tool, the Light and Spectroscopy Concept Inventory [PDF], and the fraction of class time spent on interactive, learner-centered activities. They collected data from 52 classes at 31 institutions across the U.S.

The result is not a clear, more interaction = higher learning gain, as one might naively expect.  It’s a bit more subtle:

Learning gain on the LSCI and Interactive Assessment Score, essentially the fraction of class time spent on interactive instruction. Each point represents one class with at least 25 students. (Prather et al, 2009) Our UBC result from the Sep-Dec 2010 term is shown in green.

The key finding is this: In order to get learning gains above 0.30 (which means that over the course of the term, the students learn 30% of the material they didn’t know coming in) — and 0.30 is not a bad target — classes must be at least 0.25 or 25% interactive.  In other words, if your class is less than 25% interactive, you are unlikely to get learning gains (yes, as measured by this particular tool) above 30%.

Notice it does not say that highly interactive classes guarantee learning — there are plenty of highly-interactive classes with low learning gain.

Back in September, I started recording how much time we spent on interactive instruction in our course, ASTR 311. Between think-pair-share clicker questions, Lecture-tutorial worksheets and other types of worksheets, we spent about 35% of total class time on interactive activities.

We ran the LSCI as a pre-test in early September, long before we’d talked about light and spectroscopy, and again as a post-test at the end of October, after the students had seen the material in class and in a 1-hour hand-on spectroscopy lab. The learning gain across 94 matched pairs of tests (that is, using the pre- and post-test scores only for students who wrote both tests) came out to 0.42. Together, these statistics put our class nicely in the upper end of the study. They certainly support the 0.30/25% result.

Cool.

Okay, so they learned something.  How come?

The next step is to compare student performance before and after this term’s course transformation. We don’t have LSCI data from previous years, but we do have old exams. On this term’s final exam,  we purposely re-used a number of questions from the pre-transformation exam. I just need to collect some data – which means re-marking last year’s final exam using this year’s marking scheme. Ugh. That’ s the subject of a future post…

Posted in astro 101, clickers | Tagged , , , , | 1 Comment

Wasn’t expecting Him in class

Posted on December 2, 2010 by Peter Newbury

In the #astro101 class I’m working on, we just reached the “what is life” section. Great timing, considering the new @NASA astrobiology discovery of a bacteria that, unlike every other living creature, uses arsenic instead of phosphorus in its DNA.

We were going to have a PPT slide that listed 4 “generally agreed-upon” characteristics of life

Four "generally agreed-upon" characteristics of "life". Kind of a boring PPT slide for such an intersting topic, no?

<Yawn> I suggested to the course instructor we switch it into a #clicker question, to get the students to critically think about each characteristic and then compare them to what they think “life” means:

The same content posed as a clicker question to, er, lure the students into thinking about each characteristic.

I intentionally added the last choice “E) other ______” so students could add their own ideas. The instructor and I talked about it ahead of time, and agreed that if students chose E), we’d invite them to share their ideas with the class.

Fast forward to class. We pose the question, not as a think-pair-share sequence but just inviting them to discuss it with their neighbours. Then the students voted.

Students' votes for A, B, C, D, E.

Excellent – 4 others. Wonder what they are?

“What other characteristics should a life form have?”

Then the shocker. From the back of the room comes

“God!”

In hindsight, we should have expected that! But we weren’t prepared for it. Kudos to the instructor, though: without even a pause, she replied, “Well, we’re not going to add religion and philosophy to this science class. Okay, let’s see how these 4 characteristic apply…”

The student’s answer was a great one. It told us he’d thought about the question we posed and compared it to his own knowledge, experience and beliefs. Who could ask for anything more? Be warned, though: if you want to invite your students to bring their religion into your astronomy class, be prepared – you can’t just wing it. (I did that once. Big mistake. Made me look pretty – no, make that very – ignorant.) And if you’re not familiar with the spectrum of religious beliefs in your classroom, you might want to reconsider the conversation before you start it. Why not be up front about it with your students:

Whenever people talk about the origin of life, some will undoubtedly want to include their religious beliefs. In this class, though, we’re going to stick to the scientific aspects of the discussion, the aspects that can be predicted, observed, proved or disproved by the scientific method. Now, about those scientific characteristics of life…

Posted in astro 101, clickers | Tagged , , , , | 2 Comments

Graph the graph on the graph

Posted on November 26, 2010 by Peter Newbury

I was creating a worksheet for our #astro101 class about the expansion of the Universe. If the Universe is expanding at a uniform rate, it’s about 14 billion years old. If the expansion is accelerating (decelerating), a little logic tells us the Universe must be older (younger) than 14 billion years.

I wrote the worksheet as a ranking task (“Rank the 3 models by expansion rate 1 billion years ago” and so on) using the great collection at UNL as a template. There’s also a nice graph that helps summarize the current, past and future expansion of the 3 models. This is the graph for my analogy of 3 runners, Connie (who runs at a constant rate), Alice (who accelerates) and Deena (who decelerates) practicing for a 100-metre race. The Universe version is identical except “distance” is “size of the Universe” and “cross finish line” is “now”.)

Three runners cross the finish line at the same time and going the same speed. When did they start running?

I agonized (well, that’s a bit strong but you know what I mean) over getting the students to draw the 3 curves for the uniform, accelerating and decelerating Universes or getting them to identify and label the curves given in a diagram. Fortunately, we have nice set of learning goals for the course and one says, “You will be able to sketch different scenarios for the evolution of the size of the Universe, including when the Big Bang happened and the fate of the Universe.” That clearly told me to use “Sketch…” instead of “Label…”

Great. But is “sketch” the right verb? Soon, as a colleague and I started listing all the graphing nouns and verbs we use interchangeably, I realized once again that students most likely have many interpretations of these words. My “expert” interpretation is different than their “novice” interpretation of words like

  • sketch
  • draw
  • graph (noun and verb)
  • axes
  • diagram
  • figure
  • plot (noun and verb)
  • curve
  • function

It’s not inconceivable that a student could be asked to “graph the graph on the graph” or “plot the plot on the plot”. Ay caramba!

In the end, I asked the students first to “write labels Connie, Alice and Deena next to each runner’s curve in the graph” (the one above). I figured that showed them the critical feature of the story, that all three runners crossed the line at the same time and going the same speed. Then later I asked

This graph shows the size of the Universe at each time for the uniform expansion model. Sketch the curves for the accelerating and decelerating universes. Remember that all curves must go through the current Universe  and all curves must have the same slope at that point because the slope is the Hubble Constant. Label the curves accelerating and decelerating.

If the Universe expands at a uniform rate, right now is had its current size.

The students spent about 15 minutes on the worksheet. I’m happy to report that 103 of 115 (or 90%) of the students correctly chose C) older on this post-activity clicker question

If we discover the Universe is expanding at an accelerated rate, it means the Universe is

A) younger than 14 billion years
B) 14 billion years old
C) older than 14 billion years

Posted in astro 101, clickers, interpreting graphs | Tagged , , | Leave a comment