In the current world health system, the developing countries face lack of adequate medical resources as well as the dearth of surgical experts. The inexperienced surgeons often face challenges while doing the surgeries due to lack of skills. Also, in the remote areas the medical resources are limited so the deliverability and quality of medical care becomes a concern. In order to combat this, telementoring (teletraining and telenavigation) has been used in which the trainers (experts) guide the trainees to perform the surgeries safely. Trainers give instructions verbally using voice-guided telementoring and they draw the sketches for the trainees over the screen during live video streaming of the surgical field. However, doing the surgery the surgeons (trainees) need to have excellent eye and hand coordination as they have to frequently look at the screen for the images and the tele-monitored instructions. The surgeons do not get real time depth perception of the images in 2-dimensions without being invasive. Also, the trainees have to do mapping of the locations displayed in the video with those at the actual surgical field.
The use of Augmentation Reality (AR) technology can help the surgeons in overcoming these problems by superimposing the computer generated images on the real world during intraoperative surgeries. Recently, the scientists developed a new type of augmented reality (AR) system that integrated a Microsoft HoloLens device with a three-dimensional (3D) point tracking module for medical training and telementored surgery
In this set-up, a stereo camera was integrated with HoloLens to get the 3D image of the scalpel during tele-mentored surgery. The trainer drew the trajectory to show the path and the depth of the cut which is transferred directly to trainee’s HoloLens in real time. The clinical feasibility of AR guided surgery with the tracking device was performed for skin grafting and fasciotomy where in vivo surgical operation was done on a rabbit. Using the tracking device and HoloLens, the trainee followed the animated virtual scalpel which moved along the surgical trajectories defined by the trainer to complete the surgery.
This study signifies the applicability of Augmented Technology by using HoloLens with the stereo camera which merges the virtual data (virtual surgical instruments and patient specific virtual anatomical details) to the real surgical field. The scientists designed and integrated the scalpel tracking system keeping into consideration the precision and accuracy of coordinate data as the virtual scalpel’s position directly affects the real world surgical field.
The image above shows the in-vivo skin grafting and fasciotomy surgery carried out on the thighs of the rabbit using HoloLens based telementoring system.
(a) An experienced surgeon at Ohio State University Wexner Medical Center (OSUWMC)is watching the 3D model of the thighs and drawing the optimal trajectories for skin grafting surgery and fasciotomy. (b) and (c) The 3D models of the left and the right thighs and the annotation guidance by the experienced surgeon at OSWMC are transferred back to the HoloLens device at University of Science and Technology of China (USTC) for guided skin grafting surgery and fasciotomy respectively. The black curves pointed by red arrows show the optimal surgical trajectories defined by the experienced surgeon at OSUWMC. (d) An inexperienced trainee wearing the HoloLens device is carrying out the surgical operations. (e) and (h) The HoloLens device displays the actual surgical scene superimposed with the AR scene for skin grafting (left) and fasciotomy (right) surgeries. (f) and (i) Following the virtual scalpel guidance, the trainee draws the predefined trajectories in black ink on the left and the right thighs for grafting and fasciotomy surgeries respectively. (g) and (j) The actually dissected skin tissues for skin grafting and fasciotomy surgeries.
References:
Liu, P., Li, C., Xiao, C., Zhang, Z., Ma, J., Gao, J., . . . Xu, R. (2020). A wearable augmented reality navigation system for surgical telementoring based on microsoft HoloLens. Annals of Biomedical Engineering, 49(1), 287-298. https://doi.org/10.1007/s10439-020-02538-5
As an extension of this topic, there is now the ability to help patients learn and gain a better understanding of their own bodies and conditions by using VR to view 3 dimensional images of the inside of their bodies created from scans. It also gives surgeons the same 3D experience to help them better prepare before a scheduled surgery.
Here is a 5 minute video that goes into more detail:
https://youtu.be/tZvY1a-0rpg
Hi Neill,
Thanks for your comment. VR Technology is enhancing the surgeon’s experience and helping the patients understand their own body. I came across another article which further extends the experience of surgeon trainees by integrating AI with VR for surgery. Surgeon trainees were mentored for delicate neuro surgeries using VR simulators and AI technology. This helps surgeons gain immense experience before entering the operating room for real surgery.
Here is the link to VR and AI technology used in neurosurgery:
https://www.rcinet.ca/en/2019/08/07/training-brain-surgeons-with-vr-and-ai/#:~:text=The%20fusion%20of%20artificial%20intelligence%20(AI)%20and%20virtual,more%20improvement%20before%20they%20operate%20on%20real%20people.