“I try to think about the mechanism as much as possible – it’s the very careful and most systematic “how” explanation that we should always have in mind. Still, if I see this word nonchalantly thrown around in another review of my work, I will scream. It’s important. Fundamental. So fundamental, that it should motivate the design of experiments and theory guiding them, not the speculative post-hoc interpretations we make (at least not out loud right?). Mechanism is becoming the word used by (perhaps lazy?) reviewers who don’t have anything specific to say. Ironic. That, or perhaps I’ve just been reading way too many reviews these last 2 weeks and I need to thicken up…”
– Amy Finn, PhD^
I know I said that this blog was going to be used mostly to post updates on the goings on in the lab, and this post won’t fit with that theme. But it touches on something that I’ve been thinking about a lot lately, while introducing you to the way I think, and the way I try to train grad students to think.
I think about theory, a lot. Everything I do is done with theory in mind. By theory, I don’t necessarily mean big grand theory, I mean, more specific concrete ideas about how things work (the mechanism Amy refers to), how different aspects of cognition are related to each other, and how my findings fit in with what we already know. I don’t do ‘cute’ studies that are interesting ‘just because’. I always want my work to tell us something, something bigger. This means that I think. A lot.
This makes me slower than many other researchers. I don’t jump into things quickly. And I don’t tend to write quickly either. It also means that I frustrate students. When a student comes to me with an idea for a study, especially early in their relationship with me, I usually stop them from explaining it to me part way through and ask why, why they would do whatever it is they are proposing to do. What will it tell us about anything other than the particular experiment they are describing? Usually they don’t have much of an answer. That’s why I encourage students to start with questions, not ideas for studies. Armed with a question, we can design a study to answer that question. And then we sit and think about what positive and negative results would mean, what are the possible confounds, other interpretations, and think about what the follow up studies should be given different sets of results. I almost never design one experiment at a time. And I never think about tweaking variables in a study just because they are there to be tweaked. I always want a reason, a bigger picture reason, for manipulating a variable. That’s not to say that ‘little’ variables (like ISI) aren’t important. In fact, we’re finding out in an ongoing study by Alexis Black that it (ISI) is. We found that out by accident though, and we’re now investigating it purposely, with informed ideas about why it has the effect it seems to have. Ideas that might be wrong. (Stay tuned to the blog for more on this in the near future.)
So in general, my approach to graduate training is to help students learn to think in a certain way, not to think certain things. Specifically, I want them to leave the lab approaching research in a certain way. To think big thoughts, even about ‘small’ things. The conclusions they come to, and the theories they espouse might be different than they are for me. That’s as it should be. And I try to remain open enough myself that I can learn from them as well. I’m not sure how successful I am at all of this, but it seems to be working OK. On the latter point, my own interests have been demonstrably affected by my students (see my continuing interest in gesture, for instance, which is all due to the influence of Whitney Goodrich Smith). On the former, I am heartened by the recent facebook post by Amy Finn that was the quote that lead this post.
But I am also frustrated. Frustrated for her, as I seem to have made her life much more difficult by encouraging her to think this way. Frustrated that this is so far from the standard way of working that reviewers don’t believe you when you say that contrasts are planned. And ask you to examine your data every which way, with no theoretical basis, while simultaneously chastising you for doing too many comparisons. (How doubling the number of statistical tests is a solution for too many to start with is beyond me.) And who encourage you to remove non-significant findings from a paper. You know what, I include carefully controlled variables in a study because there was reason to believe that they would affect outcomes. Sometimes I am wrong. And I think it is worth knowing that I am wrong. Especially when results from other related studies would suggest otherwise. (I have been able to include some null results before, see e.g., *Hudson Kam, 2009, so public records of my incorrect ideas do exist. But not enough.) But as we all know, null results are notoriously hard to publish. This is a topic that has been much discussed lately. And people are pushing better stats as a way to fix the problem. But that is only part of the solution. It seems to me that situating work within theoretical issues and questions that are specific enough to be meaningful (not just of the ‘hey, are these two things related’ variety) is another crucial part of the solution. But of course, this only works if we also know about failures. And understand them.
What is the point of this post? Well, it is two-fold. One, to announce that we will do blog postings of failed experiments and conditions on the blog so that there is a more public record of them, from my lab at least. Two, to make a point about the importance of thinking from a theoretical perspective. And as part of this, to inform people of how we do things in our lab. To clearly state that if you encounter a student who has worked with me, you can ask them to justify their work, to put it in a broader framework. I can assure you, they have thought about it. (And to warn people who might be interested in working with me about this. My way of working works for some people, but not for others.) We don’t have all the answers here. (If we did, I’d be out of a job.) But we’re really good at questions.
^quote used by permission
*Hudson Kam, C.L. (2009). More than words: Adults learn probabilities over categories and relationships between them. Language Learning and Development, 5, 115-145.