Is Google scholar enough for SR searching? No.

BioMedCentral has just published a paper entitled Is the coverage of Google Scholar enough to be used alone for systematic reviews?

The article is timely and asks a good relevant focussed question. The abstract is clear enough about their aims. However, when you look beyond the abstract there are some semantic problems and problems with methodology. For example, keyword searches for “erythropoietin” OR “darbepoetin” AND cancer yields a 100% recall of 36 items. However, the precision is 0.1% (36,630 articles found for 36 articles). In other words, to find the relevant 36 articles, one would have to search through ~36,700 items. Are the authors suggesting that researchers consider a precision level of .1% acceptable for the SR? Who has time to sift through that amount of information?

In any case, the reason the 36 references were found was because each was known beforehand. Searching for 36 known items is a completely different exercise when trying to find the same 36 articles scattered among a set of 36,700 citations.

The authors do not extend their logic to other searching (generalizability). In other words, it should strike them that the message they send by their title is that Google scholar is acceptable as the one-stop search tool. For their second disclosed search, the authors state as their “search string “(depression treatment placebo antidepressant) (“general practice” OR “Primary care”)” identified 16100 articles, leading to a recall of 100% and a precision of 0.09 (14 articles included in the corresponding systematic review).”

Sure all 14 articles are there, but only when doing a search after the systematic review as a post-hoc search. What do you do if you are searching Google scholar to cumulate the literature in the first place? The 14 articles are all “indexed” or “searchable” via Google scholar but only if you know what you are looking for; in other words, the authors don’t think about the difficulty associated with finding them. Just because 36 articles can be found in a massive database like Google scholar doesn’t necessarily mean you will find them. Google scholar is, like Mother Google, very large and imprecise (as the authors say .1% and .09% precision) for targeted structured searching. The extremely low precision numbers are not an insignificant matter.

In the meantime, the authors’ searches aren’t reproducible (the third problem) because we don’t know if they browsed all 36,630 articles or simply assumed one should find them given limitless time and energy. In summary, is Google scholar enough to be used alone for the systematic review? An emphatic no.

see also Google scholar | Google scholar metrics |

13 thoughts on “Is Google scholar enough for SR searching? No.

  1. Wow, Dean, I agree with your critique and then more. I’m rather shocked that this article was published as is. I would have sent it back for major revisions — at minimum calculating precision and the implications of that on search & results sorting time, to be presented alongside the recall results. To suggest that recall is basically all that matters and precision is a minor detail implies that time and budget management are insignificant. It is like saying that hacking away at a gigantic wild forest with a blunt object is as good as using a chainsaw on a few different well-maintained tree plantations, in terms of harvesting lumber. Yeah, the forest may have the same quantity of good wood in it, but…can you find it? And if so, how long will it take you?

    I think this excerpt from the discussion section is key, although I disagree with where the authors go with it:
    “Nevertheless, we should not overestimate the precision of PubMed in real life since Precision and recall of a search in a database is highly dependent on the skills of the user. Many of them overestimate the quality of their searching performance, and experienced reference librarians typically retrieve about twice as many citations as do less experienced users.”
    To me, this is very true, but my take home is that one should be sure to consult expert searchers when it’s important, not just accept that we are all hubristic but terrible searchers who must hack away at the information jungle with a blunt tool.

    To imply that GS “could even be used alone” for SR searching, and because “It just requires some improvement in the advanced search features” to become the best medical database is jaw-dropping. GS may someday become the premier medical or multi-search database even for expert seaching such as SR’s, but it has a long way to go, and the major indexing and searching infrastructure challenges are not “just” a matter of a few minor tweaks!

    This is one of the situations where I’m dying to know who the peer reviewers were and why the editor didn’t assign someone with information expertise to such an article.

  2. Totally agree!
    Furthermore the authors relied upon single string searching when anyone who routinely conducts systematic searches knows that it usually requires combining several search sets, some of which are included to demonstrate robustness even if they are a zero set of results.
    Systematic searching also entails careful mapping of concepts including potential synonyms and variant spellings etc. None of the explode, focus, or mapping functions are available and there is no way of controlling vocabulary or applying population limits.
    All the authors have really shown is that the majority of biomedical articles in peer reviewed journals can be retrieved through Google Scholar. So nothing new there!

  3. Pingback: More on using Google Scholar for the systematic review | The Search Principle: views are my own

  4. Hi,
    Don’t misinterpret our results.
    The coverage of GS is good, but its precision is low. That is all.
    The precision of the search used in the COchrane review on erythropoietin was 10 times better than our search on GS, but was still around 1%.
    What we wanted to say is that GS should make some efforts on the search possibilities.
    To Greyson, we did not say that recall is all. As clinicians, we spend a lot of time to find relevant information and we know, for sure, that the time spend must be taken into consideration.
    When we said “just requires”, we did not meant it will be easy but that the coverage is good, so let’s work on precision.
    So : is Google scholar enough to be used alone for the systematic review? An emphatic no … for the moment!
    Sincerely
    JF Gehanno

  5. Great post, and great discussion in the comments. But another issue that I think we as medical librarians are aware of to the point that it’s assumed, but others might not be, is that Google Scholar doesn’t share where its citations are coming from. Precision aside, just because the results are indexed in Scholar today doesn’t mean that they’ll be there tomorrow. This issue is briefly alluded to in the article (“issues regarding scope of coverage and currency [4,5,25]“, pg. 4), but it seems like a risky game to play. That recall of 100% could drop substantially if just one publisher backed out of their “agreement” with Google. Perhaps it seems unlikely that they would, but there could be reasons. I’m not an expert in how this works, but it seems like there’s potential for even a slight hiccup, where some publisher code accidentally blocks the Google spider from accessing a set of citations for a short amount of time, which could cause major disruption if a reviewer relying only on Scholar doesn’t get those articles in the results set that day. Only curated databases that systematically collect all their citation data in perpetuity can guarantee exactly what is available at any time. With Google Scholar unable to do that (http://scholar.google.com/intl/en/scholar/help.html#coverage), I’d argue there’s no way it could be considered the one and only source for a systematic review. What I find far more interesting and a major strength of Scholar is what can be found there that isn’t found in the standard databases. I do firmly believe that a systematic review is made stronger when Google Scholar is *one of* the tools used.

    P.S. Maybe consider moving some of this conversation (or maybe just a link back here) to the BMC reader comments for the article? Reach as many people as you can! :)

  6. I wanted to give credit where credit is due: I meant to mention that my thoughts above should be credited my MCW librarian colleagues, who brought up this line of thought during a great conversation we had at work today. Now we just need to to have the conversation with our patrons!

  7. Excellent points Amy. One of the central issues since 2004 is that Google Scholar doesn’t reveal precisely what sites it crawls and how big its database is in terms of citations. Yes, it’s helpful and yes it has its place. But we shouldn’t think it’s a curated database in the way that MEDLINE and EMBASE are curated databases.

    I also think that it’s fine for the authors of the paper to suggest Google Scholar improve its search capabilities but they simply don’t focus enough on this issue to make it their central aim.

    Thanks for making your cameo here. Cheers, Dean

  8. BMC Medical Informatics and Decision Making has open peer review, though the handling editor is not named. You can see the review process here: http://www.biomedcentral.com/1472-6947/13/7/prepub

    You can leave a comment here: http://www.biomedcentral.com/1472-6947/13/7/comments

    The last author Stefan Darmoni is an editorial advisor for the journal: http://www.biomedcentral.com/bmcmedinformdecismak/about/edboard

    (I am an Associate Editor at PLOS ONE, I formerly worked on the BMC-series journals)

  9. Thanks Matt for your post, I was about to post something similar. I looked up the reviewers, and looked at the reviews. I am a bit shocked that no librarians or specialists in information science were consulted for review. One of the reviews is extremely negative for several of the reasons already noted. It seems then, based on the author’s refutation – which mentions the concerns but doesn’t really address them – that it was accepted without further review. Wow.

  10. Pingback: What’s Wrong With Google Scholar for “Systematic” Reviews | Emerging Technologies Librarian

  11. Pingback: Google scholar for systematic reviews…. hmmmm « musings of a medical librarian

  12. what i was wondering when i read this article (and i had the very same doubts as you, before reading them, was on what place we would find the desired atricles. are they in the top 100? or somewhere in the bottom. if they are in the top 100, then they might have a little point.

    but yeah, to investigate that they’d have to browse through the complete search themselves, in stead of just combining the subject search with the known item search to check if it retrieved it.

  13. Pingback: No, Google Scholar Shouldn’t be Used Alone for Systematic Review Searching | Laika's MedLibLog

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>