The Last Hurrah! A farewell to the Science Blog, and the perils of data diving…

Greetings all,

It is with a sad note that Roger and I announce this will be our last real science blog for the foreseeable future. We are rather tied up these days with many other projects and have decided to call it a day as many other higher-profile blogs cover the same ground (and probably better so: see the side bar for examples). We have had a good run though, over the last five years, and as most blogs last less than 6 months we have done pretty well and hope we have made a useful contribution. A couple of our blogs actually got developed into published academic papers,  and in any case we certainly had some fun on the way, and so achieved our objectives. Hopefully we have given readers some interesting food for thought with our rants on this and that over the years.

The pursuit of high quality science and critical thinking that drove us to create this blog is still something we both feel strongly about, but on the less positive side, it still seems we still have a long way to go. We both come from newer university backgrounds, but our experiences suggest that the move to mass higher-education (HE) has not had the desired effect of raising scientific and artistic literacy in the population overall, but seems to have created an even more polarized educational environment. Today HE has become more of a commercially focused enterprise and science has become the catch-all phrase for any sort of inquiry that will help market a programme/course/text or product. I would place a bet that the vast majority of new graduates in my discipline have not been taught the common logical fallacies, and that more than 60% of newly qualified health care professionals in practice don’t really understand what P-values represent (see Lies, Damned Lies and Statistics if you have any nagging doubts yourself – you may be surprised). LIkewise in Science Education many newly qualified teachers still don’t seem to understand some fundamental scientific concepts.

Another example of the increasing indifference I see in scientific inquiry came to me recently when I was reviewing a set of grant applications (something I am asked to do from time to time). The ideas for the projects submitted were all excellent, and reflected innovative and practical areas needing investigation that could all provide really useful health/social outcomes. However, the approaches and methods selected to investigate the phenomena chosen seemed to indicate that the researchers were generally unconcerned with the best way to design their studies. Primarily, most researchers in the ones I reviewed couldn’t differentiate whether an inductive or deductive approach would be best to answer their particular question of interest. Several had adopted inductive methods to answer a deductive question (or the converse), some posed a deductive hypothesis and then proposed an inductive study to answer it. Several had included both inductive and deductive investigations in one study, with a mass of statistical tests in a  catch-all “lets find out everything at once” approach.

Designing complex and multi-outcome studies is certainly possible and can produce some excellent scientific work when done well, but it is a tricky business at the best of times, and as anyone who has undertaken scientific work will attest, the more complexity you add in to your study the more likely you will introduce confounding variables and have less certainty in your outcomes. In the worst cases this represents the fundamental research problem of trying to find your hypothesis in your data (also known as data diving, data fishing, data dredging, data snooping, or equation fitting).

Inductive vs. Deductive Research and Hypothesis Development

We have covered inductive thinking  and the problem of induction before on the blog (our prize is still there to be claimed)! However, a brief exploration of why it is useful to differentiate inductive/deductive approaches in scientific thinking, and how they relate to hypothesis development is worth revisiting here.  If we consider that a hypothesis is a proposed explanation of a phenomenon, it should only really come into play when we have established an explanation that we want to test, not when we are still coming up with multiple possible explanations. The following summary of the hypothetic-deductive model of scientific method may help when thinking about categorizing a line of inquiry and questions:

  1. Inductive Inquiry: Observations leads to pattern recognition, which leads to explanatory theory/theories generation, resulting in hypotheses (plural)
  2. Deductive Inquiry: A selected theory (singular) to be tested leads to hypothesis generation, leads to testing/observation, results in confirmation/refutation of theory

They both work together in a cycle of scientific inquiry, but in inductive inquiry you don’t have the hypothesis at the start (otherwise why bother – you would move straight to the deductive/testing part).

The Problems with Data Diving

Data diving/dredging seems to becoming increasingly popular in research. Probably because as we develop ever increasingly large databases and can mine them with powerful computers that can process complex statistical algorithms quickly and efficiently. It involves testing huge numbers of hypotheses about a single data set by exhaustively searching for combinations of variables that might show a correlation. The problem with it is that in proper statistical analysis we test a hypothesis with evidence (data) that was not used in constructing the hypothesis. This is critical because every data set contains some patterns entirely due to chance. In his book Bad Science, Ben Goldacre gives a nice example, in that if you randomly machine gun the side of a barn, and then go and inspect it afterwards, you might start to notice patterns in the holes that would make you think there was some sort of effect there and you could construct a hypothesis to explain it. For example possibly there was a micro-gravitational field to the left top corner of the barn making the bullets cluster there. Of course, there could have been, but every data set contains some patterns that arise entirely due to chance, so if the hypothesis is not tested on a different data set from the same population, it is impossible to determine if the patterns found are chance patterns. Another example would be if you toss a coin five times, with a result of 3 heads and 2 tails, might lead one to hypothesize that the coin favours heads by 3/5 to 2/5. As this hypothesis has been tested on the existing data set, it is confirmed! In reality, of course, the confirmation is meaningless. There are other issues with data diving too, and there is a very good wikipedia article on it here which gives many excellent examples of the main problems.

Farewell for Now…

So, it will be interesting how scientific thinking and practice develops over the next decade or so. Our sense is scientific progress will continue apace and we will eventually see the demise of postmodern nominalist and victimary thinking, as the generation of ever expanding descriptive theory has to be matched by practical outcomes and application. Science is in essence a useful way of understanding the world in that it helps us successfully predict and control events, rather than simply describe them. The volume of data on the internet is expanding at a phenomenal rate. The size of the digital universe, in terms of data being generated, is forecast by IDC to grow to a staggering 44 zettabytes by 2020, and so it will become even more important to be able to discriminate the actual information that represents of practical value to us. New and creative developments in scientific methodology and information technologies will accelerate this progress. However, we are not so sure of the future of the established university down the road in an age where knowledge and expertise are so widely disseminated.

We may of course feel the need to post the odd rant or rave on science (or other associated matters) as the mood takes us in future,  but for now we are signing off. So that’s it from us, no more typo infested and grammatically erroneous  missives for now! We wish all those who have read and contributed to the blog over the years all the best of luck.

Onwards and upwards.

Regards

Roger and Bernie

February 2014

 Reference

Goldacre B.  2008 Bad Science, Fourth Estate, London

Happy New Year: The cargo cults of science.

A belated Happy New Year to scientists and those who are interested in science everywhere from us. I am afraid I still don’t get why the greeting is Happy New Years on this side of the Atlantic. Am I missing more than one of them? Anyhow, we apologize for not getting round to our usual fun post this holiday, but we have both been swamped by work and other commitments so the blog has had to take a back seat. Believe it or not this stuff takes quite a bit of work!

Nevertheless, there was lots of good humour abound at the end of 2014 and I was pleased to note another bumper crop of Ignoble awards for 2014. My favourite has to be  the Arctic Science Prize for Eigil Reimers and Sindre Eftestøl, for heir ground-breaking study testing how reindeer react to seeing humans who are disguised as polar bears. Absolutely brilliant, and I definitely want some advice on writing my next grant proposal from them.

After my last posting (and reaction) I was discussing with some colleagues how a good understanding of the philosophy of science and scientific practice still seems so elusive for many people, and how organizations spring-up everywhere that mimic the real thing to take advantage of this. I would characterize this as a failure of education, and sadly the results in modern society are actually quite profound. We see examples in extremist behaviour and positions (often resulting in violence), the search for overly simple solutions to complex problems, the adoption of magical practices by intelligent individuals, and of course, the usual mischaracterization of science as the antithesis to religion. You only have to look around the news or the web for examples everyday, with Oprah and Dr. Oz championing the latest magical nutritional fads, or homeopath’s claiming they can cure ebola.

I find one of the most fascinating aspect of all this the alternatives to scientific inquiry and dissemination that masquerade as science. Years ago as an undergraduate student I studied social anthropology, and was fascinated by the phenomenon of cargo-cults, which represents a useful metaphor here. The term cargo cult encompasses a diverse range of practices that occur in the wake of contact with the commercial networks of technologically superior colonizing societies. The name derives from the belief that various ritualistic acts will lead to a bestowing of material wealth of some sort (cargo).

The best known example is from World War II where contact with the Japanese and late Americans military in the pacific  brought about a considerable transformation of the indigenous peoples of Melanesia (Ton, 2009). At the end of the war, the military abandoned the airbases and stopped dropping cargo by parachute to their units. Those technological products that had been seen or been available to the Melanesian islanders through the colonizing armies, disappeared or became in sort supply. In response, charismatic leaders arose amongst remote Melanesian populations and developed cults that promised to restore on their followers deliveries of these goods (such as food, arms, radios, vehicles etc.) The leaders explained that the cargo was really gifts from the gods or ancestors, and in attempts to get cargo to fall by parachute or land in planes or ships again, islanders imitated the same practices they had seen the military personnel use. They mimicked the day-to-day activities and dress styles of US soldiers, such as performing parade ground drills with wooden or salvaged rifles, and in a form of sympathetic magic, many built life-size replicas of aeroplanes out of straw and built mock airstrips complete with wooden control towers to encourage aircraft to land (as shown in the picture below).

Cargo-Cult

 

 

 

 

 

 

 

 

Even though the cargo never arrived the leaders maintained their cults for years as the social benefits of belonging to the group persisted, and they provided hope for the elusive products in the future. Today the original cults have almost disappeared but some cargo cults are reportedly still active today (Raffaele, 2006).

The term “cargo cult” represents a useful metaphor to describe an attempt to recreate successful outcomes by replicating circumstances associated with those outcomes. However, the circumstances mimicked are either unrelated to the causes of outcomes or insufficient to produce them by themselves (in the former case, this is an instance of the post hoc ergo propter hoc fallacy). The metaphorical use of the term was popularized by physicist Richard Feynman at a 1974 Caltech commencement speech, which later became a chapter in his book Surely You’re Joking, Mr. Feynman!, where he coined the phrase “cargo cult science” to describe activity that had some of the trappings of real science (such as publication in scientific journals) but lacked a basis in honest experimentation.

The term seems to have fallen out of common use today, but I would suggest we are seeing boom times for cargo-cult science today and its time to resurrect it. Recent examples include journals that purport to be scientific but are not (see our recent post of predatory publishing), many CAM therapies, naturopaths, and the growth of the Integrative science movement.

Times are tough for real scientists, and I sometimes think it would be easier to join the masses of new-age practitioners rather than keep up the struggle for good quality scientific health care, and be characterized as the allopathic dinosaur (soon to be extinct) in the face of the new age of integrative healthcare. The trouble is, I just can’t bring myself to nod sagely next time someone suggests that “sciencing the art sheds light on the meaning of universal living experiences…” Nor can I sign-on for this sort of nonsense for an easy academic life, as I fear it’s patients who actually loose out here. What the heck, maybe I’ll just go and build an effigies of Dr. Oz and Dora Kunz on the roof and see what happens…

Onwards and upwards

Bernie

References

Feynman R. (1974) Surely You’re Joking, Mr. Feynman! Adventures of a Curious Character, Richard Feynman, Ralph Leighton (contributor), Edward Hutchings (editor), 1985, W W Norton, ISBN 0-393-01921-7

Raffaele, P. (2006). “In John They Trust”. Smithsonian (Smithsonian). Retrieved Nov 26, 2009.

Ton, O. (2009). “What happened to Cargo Cults? Material Religions in Melanesia and the West”. Social Analysis 53 (1): 93–4.

Are Nurses in British Columbia Licensed to Practice Magic?

Seasons greetings to all.

An Interesting Case

This year I decided to pursue a complaint against a practitioner with the College of Registered Nurses of British Columbia (CRNBC) for a nurse who was using their Registered Nurse (RN) title to market their own private Therapeutic Touch (TT) business. To those outside of the nursing profession this might seem a little confusing, but in other countries where I have worked as a nurse it is an established professional principle that you do not use your professional qualifications/status to promote your own for-profit activities outside of your role as an RN. The reasons for this would seem fairly self-evident. For example adding RN to your name gives you some professional credibility and it would be disingenuous to use it for personal gain, such as by selling something (be it real estate, car parts, vitamins or personal life-style counselling). The CRNBC has established standards governing use of title, and has even produced  guidance for practitioners with a case example to explain this. See: Can she Use her Title? where they note that having “RN” after your name may be seen as “…important in gaining the public trust and selling more product.”

Indeed, it was this very article that incited me to make the complaint as a test case, as I came across a TT practitioners website where the title of RN was explicitly used next to the practitioners name. I used the exact format and language in the guidance for the complaint and cited the relevant “Use of Professional Titles” standard and College Bylaws (sections 4 & 8)  in the complaint, and asked that the CRNBC to ask the practitioner not to use the RN title on their website. I had been in contact with this individual on other occasions and suggested that the use of RN was inappropriate to endorse this business activity. Unfortunately, I had a completely negative response, so a simple complaint to the CRNBC on appropriate use of title seemed an appropriate next step.

At this stage it is probably worth a quick recap on what the professional regulators function is. Basically, the regulation of nurses includes licensing, the creation of professional standards, enforcement of those standards, disciplinary measures and nursing education approval. Overall, the key principles that underpin the statutory professional regulation of nursing may be considered as:

(a) maintaining the safety and quality of the care that patients receive from health professionals;

(b) sustaining, improving and ensuring the professional standards of health professionals and identifying and addressing poor practice or bad behaviour;

(c) providing systems and legislation that sustain the confidence of both the public and the healthcare professions through demonstrable impartiality; and

(d) ensuring that the integrity of health professionals is sufficiently flexible to effectively meet the different health needs of the public and healthcare approaches can adapt to future changes and demands (Hewitt 2007; ICN 2014; Walsh 2012; WHO 2002).

To ensure that regulatory standards are effective, the regulatory process itself also needs to be seen as impartial and independent from the government, the professionals themselves, employers, educators and all the other interest groups involved in healthcare (Hewitt 2007). I.e. the CRNBC should represent the interests of the public at large rather than the profession or government (this is an important point, which we will come back to). So, one would think that in terms of public protection this complaint would represent a fairly straightforward case of inappropriate use of title to support a commercial enterprise.

Another consideration here is that in this case the title was being used to help market a completely unscientific, non-evidence based and “unverifiable” practice (as indicated by the CRNBC’s own Bylaws, 2012; Section 8.04). We have discussed the merits of TT on this blog before, and it is of course, pure magic in the terms it is portrayed (remote healing using undetectable energies) with no-scientific evidence to support efficacy. A 1999 review of the physics of complementary therapies states that the existence of a “bio-field” or “bio-energetic field” directly contradicts principles of physics, chemistry, and biology (Stenger, 1999). A recent Cochrane systematic review found “no robust evidence that TT promotes healing of acute wounds (O’Mathúna et al, 2012). The American Cancer Society stated in 1999, “Available scientific evidence does not support any claims that TT can cure cancer or other diseases” (AMA, 2012).  Lastly we have the youngest person ever to get a paper published in the Journal of the American Medical Association ( the nine year old Emily Rosa) presenting a study that debunked the claims of TT practitioners in a simple study where twenty-one practitioners of therapeutic touch failed to detect her aura (Rosa et al, 2009). TT is then, complete mystical woo-woo, and fails to meet any test of scientific credibility in that 1) the underpinning theory is tautological nonsense, and b) practitioners can demonstrate no efficacy better than placebo. No respected university will touch TT practical training with a barge pole. Nevertheless, we should note it is a highly profitable enterprise  (the overheads are minimal) and many nurses do support it. The professional regulator would seem to have some responsibility to ensure practitioners are using evidence-based practices, and TT (based on current evidence) clearly falls outside that scope.

The Outcome

Anyhow, in the end the CRNBC disposition on this case earlier this year sided with the practitioner, and that they were perfectly entitled to use their RN credentials to market their TT business as they found that:

a) the regulations are not intended to apply to RNs providing education services. The CRNBC regarded this registrant as providing an educational service, as the website identified was aimed at a TT consultation service.In other words they viewed an educational service as not being a product (even though it was a commercial, for-profit enterprise)!

b)  Alternative and complementary therapies are considered within the realm of holistic nursing.

Given the CRNBC’s position on CAM in general, I was not unduly surprised with this outcome, but am concerned this represents a particularly troubling precedent as point a) represents a very narrow view of what a commercial product is, and also implies you can sell health education on anything (whether good or bad) and simply tag your RN credential to your marketing materials. Point b) is even more worrying as it represents an “anything goes” approach to the regulation of RN practice (as long as you don’t physically hurt anyone; it’s all good). This completely ignores the responsibility to make sure practitioners don’t financially or emotionally exploit the public, and tarnish the reputation and credibility of the nursing profession.

Let’s examine a comparable example. If I want to promote my business as a tarot or astrology based health practitioner, and make good money selling my advice and education to the public who believe in this. Then I can tag on my RN credential to give it an air of professional respectability, better market my magical consultation services and improve my profits. Its all good!

I can’t help but ask; how does this approach protect the public rather than the profession? The finding here does nothing to protect the public from exploitation, and seems to give carte blanche for RNs to practice magic. To illustrate this, I have just acquired a practising Witchdoctor (WD) certification from the Canadian Association of Witch Doctors (it was a tough program, and examination I can tell you)! So, with my new magical powers, apparently using my title in the following advertisement is perfectly within my scope as an RN:

Loco Advert

Click here to see the full size Locomatic Energy Inc Advert

This advert covers the same ground as the website I originally made my original complaint against (well, apart from the cat, services being free and the windsurfing). The use of the RN title to support this sort of activity is clearly nonsense, and to be frank, supporting it is a travesty of regulatory practice and represents an example of where our professional regulatory body is failing miserably. Regulators traditionally err on the side of caution, remain impartial, and represent the public interest, not those of the professionals they regulate. In this case I would argue they clearly failed to meet their regulatory obligations in this respect.

Other nursing and health professional regulators I have observed seem to do a much better job in this area. The College and Association of Registered Nurses of Alberta (CARNA) Standards on CAM, 2011 notes:

“Registered nurses cannot use the title ‘RN’ or ‘NP’ in association with the endorsement or promotion of products or services. Endorsement of a product or service occurs when a nurse uses their credentials to lend credibility to a commercial product or service (CNO, 2009). The endorsement of a product or service without providing information about other options could mislead the public and may be considered a conflict of interest.”

Likewise the College of Nursing of Ontario, Professional Misconduct Regulations, 2014 states:

“It is not acceptable for a nurse to use her/his registration status to promote personal interests such as commercial products or services. Endorsement occurs when the nurse inappropriately uses her/his credentials to lend credibility to a commercial product, product line or service. The endorsement of a product line or service without providing information about other options has the potential to mislead the public and compromise trust.”

Both clearly indicate services (such as educational consultation) are to be treated the same as products in this respect. In the UK, The Nursing and Midwifery Council Code of Conduct 2014, is similarly clear, requiring practitioners to:

  • not abuse your privileged position for your own ends.
  • ensure that your professional judgement is not influenced by any commercial considerations.
  • uphold the reputation of your profession at all times.
  • ensure that the use of complementary or alternative therapies is safe and in the best interests of those in your care.

The College of Surgeons and Physicians of BC are also much clearer on how we should consider CAM in respect to public healthcare:

“Practitioners must not misrepresent the safety or efficacy of any therapy or procedure.”

“Complementary and alternative therapies differ from conventional medicines because they are generally unproven. When an alternative treatment undergoes rigorous testing, for example in a controlled and randomized trial, then the results dictate whether the alternative treatment becomes conventional treatment, whether the unorthodox becomes accepted, and whether the unproven becomes proven. Assertions, speculations, and testimonials do not substitute for scientific evidence.”

I did briefly pursue this CRNBC disposition on this case with the BC Health professional Review Board (HPRB), but have dropped it, as there is really nothing they can do in such a case as the HPRB are focused on interpretation of process and the law and are very unlikely to intervene in what they see as “professional jurisdiction” unless there is clearly evident harm arising. Astonishingly, it appears guidance published in CRNBC newsletters and advice has no real legal grounding anyhow, so can be completely different from interpretation of the actual standards by the disciplinary board. This doesn’t seem to reflect a well coordinated organization to me though. So, my next action is another formal approach to the CRNBC (I have raised this issue with them before) requesting they review their regulations and toughen them up in this respect.

Finally, although TT is generally harmless, we should be reminded these are serious concerns and not the trivial issues they are often taken as. Firstly, financially exploiting vulnerable people is a serious issue. Secondly, where the public are advised to consider alternatives to conventional evidence-based treatments, life-threatening results occur, as exemplified by the recent cases of the parents of two children with acute leukaemia seeking CAM remedies, and with people being advised to take homeopathic vaccines rather than actual vaccines. If we continue with this  approach to regulation and CAM practices, and allow them to be marketed with implied professional support, sooner or later deaths or injury will result from people In BC choosing CAM over medical alternatives on the advice of a nurse using their professional title to support their CAM practice. Effective regulation can help prevent this.

So, are nurses in BC licensed to practice magic? What do you think…

Bernie

References

American Cancer Society (2012) Therapeutic Touch.  Retrieved 20 October 2014.

Hewitt, P. 2007. Trust, Assurance and Safety – The Regulation of Health Professionals in the 21st Century. London: H.M. Stationary Office

International Council of Nurses (2014). Regulatory Board Governance Toolkit.  http://www.nurse.or.jp/nursing/international/icn/report/pdf/2014m/04-04.pdf. Retrieved July 29, 2014.

O’Mathúna, D. N. P.; Ashford, R. L. (2012). O’Mathúna, Dónal P, ed. Cochrane Database of Systematic Reviews (Online) 6: CD002766

Rosa, Linda; Rosa, E; Sarner, L; Barrett, S (1998). “A Close Look at Therapeutic Touch”. JAMA 279 (13) 1005–10

Stenger, Victor (1999). “The Physics of ‘Alternative Medicine’ Bioenergetic Fields”. The Scientific Review of Alternative Medicine. 3(1) 79-84

Walsh, P. 2012. “Health Professional Regulation.” AvMA Medical and Legal Journal 18(3): 3–4.

World Health Organization (WHO). 2002. Nursing and Midwifery: a Guide to Professional Regulation. Cairo: WHO Publications Eastern Mediterranean Regional Office.

Other Relevant Sources

1. The CRNBC has a self-identified mandate to “protect the public through the regulation of registered nurses and nurse practitioners: https://www.crnbc.ca/crnbc/Pages/Default.aspx

2. Registrants are required to use their title(s) in ways that comply with the: CRNBC Standards of Practice. See: https://www.crnbc.ca/Standards/Lists/StandardResources/343AppropriateUseofTitlesPracStd.pdf

3. CRNBC October 2013 case study “Can she use her RN title?” which was subsequently amended – current version is at: https://crnbc.ca/Standards/resourcescasestudies/beinganurse/selfemployment/Pages/marketing.aspx .