When the Research Doesn’t Exist: Finding creative solutions to research gaps in sport (Problem Solving)

The Background

It is my opinion that tactical knowledge is the most complex performance skill to teach. You want to skate better? Go to a rink and skate. You want improve your hitting? Go to a field and hit baseballs. It may not be easy but it’s certainly simple. Now obviously, the type of coaching you receive is a factor and, at some point, anthropometric factors and talent would limit you but ultimately, the process is very straightforward. By contrast, tactical understanding is a nebulous concept. As a coach or an athlete, you’re not only training decision-making but the factors that lead into making a decision.

In discussions with coaches across BC, the complaint I hear most often is that an official “doesn’t understand the game” or “has no game-sense”. There are a number of situations that could trigger this complaint: an official might have called a penalty that was perceived as unnecessary, they might have been in the wrong position, or they may have responded to a coach or player in a way that was perceived to be inappropriate. Regardless of the specifics, the core complaint is that the official lacks a comprehensive tactical knowledge of the game of hockey. In conversations with decision-makers in elite officiating development programs at the NSO and Major Junior (U21) level, the most frequent reported reason for an official’s release from the program/league staff was lack of “game sense” or “hockey IQ”, terms that are synonyms for tactical knowledge.

An ongoing source of frustration for me has been that our our key decision-makers and trainers have no idea how to train tactical knowledge. If an official is lacking in tactical knowledge, they are told “go and fix that”. No resources, no training, no clear direction on the how. So, in practice, an official either fixes it by stumbling around in the dark (so to speak) and progresses, or they crash out. As people within the officiating discipline push to create a culture of coaching, these systems gaps (and really a lack of desire to close them) are a huge obstacle that must be overcome.

What I discovered as I dove into the research is that there is not a clear methodology for teaching tactical understanding. More specifically, there is not a methodology that has been tested by empirical research and is “proven” to improve competition performance. There are plenty of studies that attempt to measure or improve tactical knowledge in various sports but none that actually compare the teaching methodology to on-field/court/ice performance. So a player might improve their tactical knowledge (as measured through an out-of-competition test) over the course of an intervention but there’s no evidence that improving tactical knowledge (as measured by the study) actually improves performance in competitions.

Planning the Intervention

So I am left with a lot of interesting information and ideas but no way of knowing what will work. Fortunately(?), due to the pandemic, I have an endless amount of time between now and the next ”meaningful” competition. For my officials, this will likely be a Hockey Canada national championship in early 2022 and/or the 2021-22 slate of IIHF world championships. So I am going to embark on a quasi-experiment to test the efficacy of one of the methods proposed by the body of research.

I won’t detail the entire methodology here but essentially, the intervention centered around video-based debriefs of volleyball players on the Spanish national team. (https://doi.org/10.1177/0031512516650628) Each debrief followed three phases: first, the player watched the clip three times to try and put themselves back in the game context of the action, with help to remember the score, the moment in the set and match. Second, the player scored their performance on a scale of 0-10 with 0 being totally incorrect, and 10 being totally correct. Third, the athlete analyzed and evaluated their decision-making process, including causes and reasons for the decisions they made.

The protocol itself was quite simple. The players were asked only one question: “what were you thinking about while playing that point?” The authors coded the athletes’ response to this question according to a three-level conceptual framework, validated by previous research. Level 1 was conceptual content of the response, or how well athletes were able to communicate the goal, the situation, and the action they undertook. Level 2 was conceptual sophistication of the response, or how well the athlete understood their action in the context of their own skill, their teammates, opponents, and the game as a whole. Level 3 was conceptual structure, or how well athletes were able to link concepts to each other. This allowed the authors to analyze responses according to a variety of variables that would indicate level of tactical knowledge.

What was missing from this study was a linkage to the athlete’s on-court performance. While all the players were performing at the requisite level for inclusion into the Spanish National Team, it would have been useful to have in-game performance measured alongside the intervention. Despite the observed improvements in athletes’ expression of tactical knowledge during debriefs, there is no mechanism for measuring whether their tactical decision-making improved There are several possible avenues for this: their coaches or an external expert could have evaluated the athlete’s performance according to a rubric or the individual athletes’ performance could have been measured through kills, attack errors, or hitting efficiency, as the situations recorded and evaluated pertained specifically to attacking plays.

Despite this limitation, the methodology seems plausible and fits with the tools I have at my disposal. Currently, video feedback is the primary vehicle for tactical training in officiating. Moreno’s research not only validates the concept but provides a useful structure for achieving tactical-cognitive improvements in elite sport. Given officiating’s historical overreliance on the “eye test” as a method of game performance evaluation, this framework will be vital to introducing empirical validity to performance evaluation in officiating. I have adapted the rubrics from the study to an officiating-specific context so that I can measure conceptual content and conceptual sophistication. The conceptual structure doesn’t need to be adapted because it’s simply measuring how categories/concepts are linked together in the officials’ verbal responses.

Now What?

Obviously, the objective of the intervention is to improve the tactical knowledge of the officials in my program so that they can challenge at the top level of national and international competition. Through the Winning Style of Play assignment, I’ve produced categories of statistical analysis for tactical decision making. Although the benchmarks need further study to be considered valid, I feel very comfortable using the categories of evaluation that were developed based on our national standards and my “expert” judgement.

What this means is that I have the ability to look at past game tape (which I started doing as I was developing the WSP) and grade officials according to the WSP categories. Then, when we return to the ice in September/October 2021 (fingers crossed) and begin regular season play, I will be able to evaluate if their tactical decision-making has improved according to those metrics. It’s not foolproof but it’s also not a fully authorized study and I’m running this all by myself, so I’m okay with balancing efficacy with realism.

Ultimately, I am predicting this evidence-backed intervention will produce an improvement in the on-ice displays of tactical knowledge of the officials in my program. I am also hoping that by going through this process, I will learn more about methodologies for teaching tactics and will also collect data that will help validate my WSP statistics. Finally, this may be an avenue for further research and lead to the opportunity to run a study that will be able to validate these methodologies in a hockey officiating context. When I do a lot of research on a narrow range of topics, I see the same names crop up over and over again… perhaps that could be me in the future?

Leave a Reply

Your email address will not be published. Required fields are marked *