Impact & Peer Review: the problem with journal rankings as a source of evidence weighting

Last week, Roger Watson editor supremo of JAN published a blog piece on the importance of impact factors for academic journals in which he noted “The world of editing seems to be divided into two camps: those who agree with the use of impact factors and those who don’t.”

To be honest I am not sure I would agree with that. Apart from the fact this seems a thinly disguised excluded-middle/false-dichotomy argument (along the lines of “your are either with us or with the terrorists”) I think there are both benefits and pitfalls to journal rankings.

Firstly we should clarify what they are. A citation index is an index of citations between publications, allowing the user to easily establish citation links between documents. The first of these were created for the legal profession such as Shepard’s Citations (1873). The development of these (and particularly the advent of modern computer systems and databases) has supported the development of Impact Factor rankings. The impact factor (IF) of an academic journal is a measure reflecting the average number of citations to recent articles published in the journal for the two preceding years (although it can be for any specified period of time). It is frequently used as a proxy for the relative importance of a journal within its field. It was devised by Eugene Garfield and IFs are calculated yearly for those journals that are indexed in the Journal Citation Reports database. For example, if a journal has an IF of 2 for 2012, then its papers published between 2010-2011 received 2 citations each on average in papers published in 2012 in journals covered in the index. Sounds straightforward and objective enough, although I suspect if you have read this blog for any length of time I expect you hear a “but” coming…

1. It is a bit of a blunt tool. Good science is not always published in the peer-reviewed literature covered in the database. Especially today, much activity at the forefront of research is reported first in blogs, websites and at conferences.

2. The peer review process that results in publication is not exactly objective. Editors have a vested interest in circulation (despite claims that good science trumps circulation). I have been on the editorial board of a couple of journals for over 10 years and can attest to this wholeheartedly. Editorial boards generally support research and papers that fit within the current disciplinary paradigm (even when papers criticizing the current wisdom based on good science or argument arrive). Also, as publication is a competitive business papers by established authors in the field will generally receive higher support for publication than those of less well-known researchers. Naturally, anything written by Stephen Hawking is more likely to get published than a paper by an unknown physics prof. This tends to lead to a bit of a beauty-contest mentality and we inevitable end up with top ranked IF journals publishing the same sorts of papers from the same range of well-known authors. Many would argue in an academic meritocracy this is how it should be, but from the perspective of heterodoxy it is more problematic.

However, there are other problems with the peer review process. In an effort to establish if my own peer reviewing was within the normal range I once asked a journal editor what their rejection rate was. I got no response, and few editors will divulge this statistic openly (for most well-known journals it is well above 70%). Reviewers may take a position against a paper because it is bad, or because of their own methodological bias, a dissenting view, or simply because they were too busy to really read it but didn’t like the abstract. In my own academic career I have had one paper rejected at one journal (stating it was “not of interest” to readers) only to get it published in another and then get into their top 20 papers downloaded.

Some journals will accept (subject to amendments), a paper that has excellent new findings and methods but requires a few grammatical corrections. Other high IF Journals who have their pick of established authors may not, and will reject anything not in A1 publication ready form on submission. I have had colleagues who have had papers rejected from journals following two excellent reviews, with only minor typo changes requested. However, these same journals have accepted and published pretty awful examples of the English language from well-known people in the field! I have frequently received two different peer reviews for a paper, one saying it was rubbish whilst the other claiming it was the greatest thing since sliced bread. A single negative review from an established reviewer who holds some authority and power with the editorial board will likely sink any paper.

My experience is for the majority of researchers, even with sound work your chances of getting a good peer review in a journal are very much dependent on the luck of the draw. Reviewers are unpaid, busy academics, and if your paper is sent to one who is pressed for time, they may not give it their best consideration. In essence your chances of getting a paper into print in the top journals is competitive, but with the rules for the competition being something of a dark-art. So much for the academic meritocracy.

3. The impact factor measures citations not good science. If I manage to get an erroneous or controversial piece published it may well get widely cited. For example. If I manage to get a paper published that notes that Florence Nightingale had a little  known punk period in her adolescence in which she died her hair green to infuriate her parents, it may get a huge amount of citations despite the fact it is completely fictitious (as far as I am aware). The Davenas et al. (1988) paper proposing the memory of water theory (published in Nature no less) that was widely cited but subsequently discredited, is an excellent example.

4. Lastly, the impact factor seems to have little to do with the professional impact of the journal (at least in my field). I regularly make it a point to always ask new nurses I meet “In all honesty what professional/academic journals do you actually read?” To date less than 3% ever cite the Journal of Advanced Nursing or other high impact journals. So what impact are we really measuring here, impact amongst the nursing academics who make up the minority of our profession?

At the end of the day, both the peer review process and impact factor although flawed, are probably the best tools we currently have to promote good quality science. The fact that that some duffers like Nursing Science Quarterly get taken off the citation index (since 2009) shows that at least there is evidence of some rigour in the process. However in the new age of Web 2.0 things could change. It will also be interesting to see what impact of the huge shift to electronic publishing and pay-for-publication (usually around $2000 a paper) open access journals will have on the traditional journals.

Happy Canada Day to all; time to hit the BBQ and pass out the maple cookies!

Bernie

 Reference

Davenas, E., Beauvais, F., Amara, J., Oberbaum, M., Robinzon, B., Miadonna, A., Belon, P. (1988). Human basophil degranulation triggered by very dilute antiserum against IgE. Nature, 333(6176), 816-818.

 

 

 

 

 

Bernie’s Philosophy of Science eBook is Alive!

Apologies for the lack of activity on the site for the last few weeks, but I had been putting the final touches to my ebook and Roger is off on his travels. Anyhow, the ebook is now published and so in a bit of unabashed and shameless self-promotion here are the details (cue the fanfare)…

So what is it about, you may well ask? In the book I present a pragmatic approach to the philosophy of health science (particularly nursing) that underpins evidence-based practice (EBP), including an exploration of research methodologies. It covers the history of scientific inquiry and foundational principles underpinning science as applied in the field of professional healthcare. I have used some provocative arguments to challenge some of the constructivist and postmodern approaches to nursing knowledge. There is a chapter exploring some key examples of non-science, pseudoscience, and simply bad-science that continues to pervade modern healthcare research. Basically, if ever you want some good arguments to challenge postmodern or relativist philosophical positions in healthcare, this book should provide you with plenty. It also includes an updated version of our good science detection kit, and an excellent foreword by Roger (although I fear endless claims of “it must be your round” when I am next in the UK as a result of that).

The book is available from a range of ebook sellers and I will update the list as they put it out ( see details on ebook readers at the end of the post) . You can search for it by my name, the title, or the ISBN: 9780991984602.

Sources 

It has been an interesting journey on the road to publication, and the book should be found at around $9 (which was the price I agreed; although, strangely some vendors seem to have added a few dollars, so shop around). I have had some excellent reviews and some dissent with the ideas presented  (but that is always the way with controversial issues). Anyhow, I hope people will keep an open mind while they read it and enjoy the book, finding something of interest. Do let me know your thoughts on the book in the blog. I welcome all constructive criticism, ideas, commentary and feedback. I am still finding the occasional typo now and again too, so apologies, I will amend, but hey it’s only $6!

A publicity flyer PDF for the book is available here.

If anyone has any trouble finding a copy, or format issues (I can supply any ebook format), please e-mail me; likewise any educators who would like an inspection copy.

Regards

Bernie

 P.S. Additional eBook Reader Information

The ebook was developed for the Apple iPad, but I have been testing various free ebook readers with the book and interestingly found some differing results. Despite adhering to all of the latest publishing standards for ebooks it seems full standardization is some way off in the electronic publishing world. I thought the following might be of use (click on the name of the application to download it):

  • Apple iBooks: displays perfectly – available via iTunes – best choice for iPad readers
  • Kindle Reader: displays fine, but loss of coloured fonts, lines and formatting of titles, in most versions. Readers may need to resize text in their Kindle to get optimum page layout.
  • Sony Reader: Displays fine- but the application is a bit slow to run on some PCs .
  • Kobo: displays perfectly The reader is available here: http://www.kobo.com/desktop
  • Lucidor: the book displays perfectly but some Windows users have experienced installation issues with this reader (it also needs Firefox to be installed first)
  • Adobe Digital Editions: Displays fine, but it seems the font size is not resizable in the application, which can mean fine sized print on some hi-resolution monitors
  • Calibre: a fine free ebook reader, and the book displays perfectly in it
  • ePubReader: displays Ok but format display issues with chapter titles (does not follow font size coding). This application also needs Firefox installed and works through it.
  • Aldiko EReader: Not tested on this Android platform but uses epub standards so should be OK. Let me know if anyone has tested it on this reader.

I hope this is helpful.

Bernie