When the Research Doesn’t Exist: Finding creative solutions to research gaps in sport (Problem Solving)

The Background

It is my opinion that tactical knowledge is the most complex performance skill to teach. You want to skate better? Go to a rink and skate. You want improve your hitting? Go to a field and hit baseballs. It may not be easy but it’s certainly simple. Now obviously, the type of coaching you receive is a factor and, at some point, anthropometric factors and talent would limit you but ultimately, the process is very straightforward. By contrast, tactical understanding is a nebulous concept. As a coach or an athlete, you’re not only training decision-making but the factors that lead into making a decision.

In discussions with coaches across BC, the complaint I hear most often is that an official “doesn’t understand the game” or “has no game-sense”. There are a number of situations that could trigger this complaint: an official might have called a penalty that was perceived as unnecessary, they might have been in the wrong position, or they may have responded to a coach or player in a way that was perceived to be inappropriate. Regardless of the specifics, the core complaint is that the official lacks a comprehensive tactical knowledge of the game of hockey. In conversations with decision-makers in elite officiating development programs at the NSO and Major Junior (U21) level, the most frequent reported reason for an official’s release from the program/league staff was lack of “game sense” or “hockey IQ”, terms that are synonyms for tactical knowledge.

An ongoing source of frustration for me has been that our our key decision-makers and trainers have no idea how to train tactical knowledge. If an official is lacking in tactical knowledge, they are told “go and fix that”. No resources, no training, no clear direction on the how. So, in practice, an official either fixes it by stumbling around in the dark (so to speak) and progresses, or they crash out. As people within the officiating discipline push to create a culture of coaching, these systems gaps (and really a lack of desire to close them) are a huge obstacle that must be overcome.

What I discovered as I dove into the research is that there is not a clear methodology for teaching tactical understanding. More specifically, there is not a methodology that has been tested by empirical research and is “proven” to improve competition performance. There are plenty of studies that attempt to measure or improve tactical knowledge in various sports but none that actually compare the teaching methodology to on-field/court/ice performance. So a player might improve their tactical knowledge (as measured through an out-of-competition test) over the course of an intervention but there’s no evidence that improving tactical knowledge (as measured by the study) actually improves performance in competitions.

Planning the Intervention

So I am left with a lot of interesting information and ideas but no way of knowing what will work. Fortunately(?), due to the pandemic, I have an endless amount of time between now and the next ”meaningful” competition. For my officials, this will likely be a Hockey Canada national championship in early 2022 and/or the 2021-22 slate of IIHF world championships. So I am going to embark on a quasi-experiment to test the efficacy of one of the methods proposed by the body of research.

I won’t detail the entire methodology here but essentially, the intervention centered around video-based debriefs of volleyball players on the Spanish national team. (https://doi.org/10.1177/0031512516650628) Each debrief followed three phases: first, the player watched the clip three times to try and put themselves back in the game context of the action, with help to remember the score, the moment in the set and match. Second, the player scored their performance on a scale of 0-10 with 0 being totally incorrect, and 10 being totally correct. Third, the athlete analyzed and evaluated their decision-making process, including causes and reasons for the decisions they made.

The protocol itself was quite simple. The players were asked only one question: “what were you thinking about while playing that point?” The authors coded the athletes’ response to this question according to a three-level conceptual framework, validated by previous research. Level 1 was conceptual content of the response, or how well athletes were able to communicate the goal, the situation, and the action they undertook. Level 2 was conceptual sophistication of the response, or how well the athlete understood their action in the context of their own skill, their teammates, opponents, and the game as a whole. Level 3 was conceptual structure, or how well athletes were able to link concepts to each other. This allowed the authors to analyze responses according to a variety of variables that would indicate level of tactical knowledge.

What was missing from this study was a linkage to the athlete’s on-court performance. While all the players were performing at the requisite level for inclusion into the Spanish National Team, it would have been useful to have in-game performance measured alongside the intervention. Despite the observed improvements in athletes’ expression of tactical knowledge during debriefs, there is no mechanism for measuring whether their tactical decision-making improved There are several possible avenues for this: their coaches or an external expert could have evaluated the athlete’s performance according to a rubric or the individual athletes’ performance could have been measured through kills, attack errors, or hitting efficiency, as the situations recorded and evaluated pertained specifically to attacking plays.

Despite this limitation, the methodology seems plausible and fits with the tools I have at my disposal. Currently, video feedback is the primary vehicle for tactical training in officiating. Moreno’s research not only validates the concept but provides a useful structure for achieving tactical-cognitive improvements in elite sport. Given officiating’s historical overreliance on the “eye test” as a method of game performance evaluation, this framework will be vital to introducing empirical validity to performance evaluation in officiating. I have adapted the rubrics from the study to an officiating-specific context so that I can measure conceptual content and conceptual sophistication. The conceptual structure doesn’t need to be adapted because it’s simply measuring how categories/concepts are linked together in the officials’ verbal responses.

Now What?

Obviously, the objective of the intervention is to improve the tactical knowledge of the officials in my program so that they can challenge at the top level of national and international competition. Through the Winning Style of Play assignment, I’ve produced categories of statistical analysis for tactical decision making. Although the benchmarks need further study to be considered valid, I feel very comfortable using the categories of evaluation that were developed based on our national standards and my “expert” judgement.

What this means is that I have the ability to look at past game tape (which I started doing as I was developing the WSP) and grade officials according to the WSP categories. Then, when we return to the ice in September/October 2021 (fingers crossed) and begin regular season play, I will be able to evaluate if their tactical decision-making has improved according to those metrics. It’s not foolproof but it’s also not a fully authorized study and I’m running this all by myself, so I’m okay with balancing efficacy with realism.

Ultimately, I am predicting this evidence-backed intervention will produce an improvement in the on-ice displays of tactical knowledge of the officials in my program. I am also hoping that by going through this process, I will learn more about methodologies for teaching tactics and will also collect data that will help validate my WSP statistics. Finally, this may be an avenue for further research and lead to the opportunity to run a study that will be able to validate these methodologies in a hockey officiating context. When I do a lot of research on a narrow range of topics, I see the same names crop up over and over again… perhaps that could be me in the future?

Problem-Solving: Connecting and training meaningfully during a pandemic

Nobody’s routine has escaped the grasp of the Covid-19 pandemic. In hindsight, it seems inevitable, but the nature of sport is that you prepare for the unexpected. So, when that door finally slammed closed, the disappointment within our group was immense. Somewhat predictably, the malaise of a canceled 2019-20 season carried through the off-season and into what would usually be our “preparation period” ahead of the new season. Most of my athletes were confined to their living rooms or garages with a smattering of outdoor activities mixed in. As the pages of the calendar flipped closer to what would ordinarily be the start of training camps and exhibition play, it was impossible to ignore the fact that most ice rinks didn’t even have a date to re-open and going to a public gym felt like an immeasurable risk. Of course, everyone was dealing with a similar reality. What made my interactions that much more challenging was the fact that my group is spread out across the province: from Comox to Vancouver to Prince George to Castlegar and all points in between. Early in the summer, the leaders within our athlete group were great about organizing weekly check-ins and short Zoom calls that were more about personal well-being than anything sport-specific. However, as the summer dragged on and the worries compounded (sporting, educational, economic, etc.), enthusiasm for that waned as well. You can only ask someone “how was your week?” so many times when the answer is “I go to work and spend the rest of the time at home”.

Of course, hanging over all this was the spectre of Hockey Canada canceling the U18 National Championships in November (which they did in late August) and the International Ice Hockey Federation canceling their slate of 2020-21 World Championships. Meanwhile, we still had no firm date for a return-to-hockey and motivation was at an all-time low. In officiating, at the level we’re talking about, with female athletes, it’s all about performance at National tournaments as a springboard to the International level. That is compounded by the high level of uncertainty that accompanies every trip to the arena: will the game go ahead? Will we arrive to find that players are sitting out due to positive tests? How many fit players will each team have? This uncertainty continues after the game: every cough is overanalyzed, the public health warnings are watched carefully; however unlikely the possibility of exposure might be, the risk is still there.

So in these circumstances, my coaching staff and I set out to figure out how to engage our athletes in a meaningful way. For context, I’m usually on the road at least two weekends per month. So under normal circumstances, I see everyone in our group a couple of times a month; sometimes I go to them, other times they come to me. Now, not only is it logistically more difficult but also our program’s budget has been slashed as our PSO grapples with the financial uncertainty of COVID. So, like everyone else, we were going to be pushed into remote-delivery for the majority of our programming. While our athletes want to be engaged and want to keep training, they have asked us to provide them with a plan and a pathway through this pandemic that keeps them accountable without completely depleting their motivation.

With that in mind, I got together with our coaching staff to answer two key questions:

  1. What skills could be credibly trained (and monitored) in a remote-delivered environment?
  2. What activities or learning experiences do we “not have time for” under normal circumstances that we could prioritize now?

After some discussion, we decided to adopt three programming directives for the 2020-21 season:

  1. Treat the 2020-21 season as an “extended specific preparation phase” for the 2021-22 season, from a YTP perspective. Our rationale for this is that even though there will be games played, there is virtually no chance of having “meaningful” games at an elite level. Therefore, why restrict ourselves to the demands a normal season when we could think of this as a 10-month preparation phase for the 2021-22 season? Let’s use that flexibility to adopt a long-term mentality, do some experimentation, and see if we can learn something valuable.
  2. Increased focus on tactical development. Fortunately, every game in which our officials participate is available via third-party video. One of the biggest challenges for the athletes that enter our program is an underdeveloped tactical understanding of the game. In 2020-21, a lighter game schedule allows us to spend more time doing video breakdown with our athletes and because the games that are being played are at a lower level, it provides them with a more forgiving environment to practice implementing that tactical understanding.
    • This also applies to things like communication strategies and mental control. We can introduce new concepts and allow the athletes time to practice and find what works for them without the looming spectre of a major competition, by which time we need to have them settled into a routine.
  3. De-prioritize physical training. For context, physical training is usually our number one priority. So we obviously aren’t tossing it out altogether but rather moving it down the list. With all the uncertainty, and the associated physical and mental health challenges, setting aggressive athletic targets for our group just doesn’t seem useful. Some of our group is continuing to set those goals for themselves and we are supporting them in doing so but we are “taking our foot off the gas” with others, particularly our NextGen athletes.

As we are currently in-progress with implementing these programming directives, I feel like a more detailed breakdown will be the subject of a future blog post. But I also put on my educational hat and thought about how best to deliver these concepts. After much discussion with both coaching and educational colleagues, we settled on the following structure:

  1. Weekly 45-minute webinars. Given how much time everyone is spending on Zoom or in front of screens, the consensus was that this is the longest we could reasonably expect anyone to focus their attention.
  2. A 6-week cycle that addresses physical training, tactical understanding, and mental skills.
    • Week 1: Ask each athlete to commit to one physical training intervention that they could accomplish in the next six weeks. It could be goal-oriented but it could also be experimental (i.e. hypothesize-and-test). This is an opportunity to consult with our coaching team and plan out what that intervention will look like.
    • Week 2: We provide a specific mental preparation strategy and task them with implementing this in their game environments. This allows everyone a couple of weeks to implement, tweak, and report back on whether this was a useful strategy or not.
    • Weeks 3-4: We provide a specific tactical intervention that we have identified for the group as a whole and task them with working on this in their game environments. Each week will provide an opportunity to look at new video and discuss implementation.
    • Week 5: This is an opportunity to debrief the success of the mental preparation strategy that we introduced in Week 2, as well as the tactical intervention from Weeks 3 & 4.
    • Week 6: This is the conclusion of our cycle as well as Week 1 of the next cycle. This is where we check in on the physical intervention that they committed to in Week 1, assess the success of the intervention, and commit to a new intervention over the next six weeks.

Any suggestions on how to engage a group when your in-person hours have been cut to the bone (and then some) as well as keeping motivation up when there’s nothing to work towards?