- Jean-François G, Laetitia R, Stefan D. Is the coverage of google scholar enough to be used alone for systematic reviews. BMC Med Inform Decis Mak. 2013 Jan 9;13(1):7.
BioMedCentral has just published a paper entitled Is the coverage of Google Scholar enough to be used alone for systematic reviews?
The article is timely and asks a good relevant focussed question. The abstract is clear enough about their aims. However, when you look beyond the abstract there are some semantic problems and problems with methodology. For example, keyword searches for “erythropoietin” OR “darbepoetin” AND cancer yields a 100% recall of 36 items. However, the precision is 0.1% (36,630 articles found for 36 articles). In other words, to find the relevant 36 articles, one would have to search through ~36,700 items. Are the authors suggesting that researchers consider a precision level of .1% acceptable for the SR? Who has time to sift through that amount of information?
In any case, the reason the 36 references were found was because each was known beforehand. Searching for 36 known items is a completely different exercise when trying to find the same 36 articles scattered among a set of 36,700 citations.
The authors do not extend their logic to other searching (generalizability). In other words, it should strike them that the message they send by their title is that Google scholar is acceptable as the one-stop search tool. For their second disclosed search, the authors state as their “search string “(depression treatment placebo antidepressant) (“general practice” OR “Primary care”)” identified 16100 articles, leading to a recall of 100% and a precision of 0.09 (14 articles included in the corresponding systematic review).”
Sure all 14 articles are there, but only when doing a search after the systematic review as a post-hoc search. What do you do if you are searching Google scholar to cumulate the literature in the first place? The 14 articles are all “indexed” or “searchable” via Google scholar but only if you know what you are looking for; in other words, the authors don’t think about the difficulty associated with finding them. Just because 36 articles can be found in a massive database like Google scholar doesn’t necessarily mean you will find them. Google scholar is, like Mother Google, very large and imprecise (as the authors say .1% and .09% precision) for targeted structured searching. The extremely low precision numbers are not an insignificant matter.
In the meantime, the authors’ searches aren’t reproducible (the third problem) because we don’t know if they browsed all 36,630 articles or simply assumed one should find them given limitless time and energy. In summary, is Google scholar enough to be used alone for the systematic review? An emphatic no.