Extended reality in cardiovascular surgery

Extended reality refers to three modalities that blend the physical and virtual worlds—virtual reality, augmented reality, and mixed reality. These modalities create immersive experiences for users, often via wearable headsets and handheld controllers. Although VR, AR, and MR often overlap in functionality, the modalities offer different levels of immersion.

Virtual reality offers a digitally simulated experience of reality based on imaging. Haptic devices allow users to navigate 3D models in a computer-generated, multisensory, virtual environment.

Augmented reality superimposes digital content, like 2D or 3D images, onto the user's native environment. While VR is restricted to a fully virtual environment, AR allows users to interact with a real environment combined with digital projections like computer-generated images or holograms.

Mixed reality provides the most interactive link between the physical and virtual worlds. The native environment syncs with digital projections through a computer system, linking reality and 3D or 4D models. This means surgeons can see their real surgical view and 3D models of a surgical case, enabling interactions with both the native and virtual environment simultaneously.

Why is it useful?

XR has the potential to benefit cardiovascular surgery in many ways. It can help with surgeon training, surgery planning, surgery execution, and postoperative pathways. In terms of training, XR can provide an engaging learning experience by allowing the user to interact with education materials in real time. This allows users, especially older individuals, to more easily work at their own pace, obtain practice repetitions in a shorter period of time, and better retain the lessons taught.

Next, XR can help before procedures. Traditional 2D imaging already plays a vital role in diagnostics and planning. But surgeons may benefit from more accurate imaging that XR can provide, like 3D CTs, virtual bronchoscopies, and 4D echocardiograms. VR- and AR-generated images offer a better interpretation of anatomic structures, and MR enables users to analyze and edit these virtual anatomic objects without obstructing the visual view, which allows better identification of the relationship between organs. XR modalities also enable a surgeon to verify whether a tumor is inside the anatomic borders of a particular segment when planning for lung resections, which can lead to a better selection of patients who are suitable for segmental anatomic resections.

XR can then help during procedures. The technology allows users to overlay preoperatively constructed 3D models onto the real surgical field. This can provide detailed anatomic and physiologic information, as well as intraoperative guidance to support a procedure and make it more accurate, safe, and efficient. The use of XR, specifically holograms, can also result in less radiation and be used in instances where a patient can’t receive contrast dye. These advantages can contribute to reduced procedure times and reduced risk of complications.

After surgery, XR can help distract patients during a stressful or painful rehabilitation. Postoperative recovery and rehabilitation are greatly influenced by psychological stress and post-op pain, and VR has been shown to help manage pain via cybertherapy as shown in one study published in The Annals of Thoracic Surgery. In addition, a VR-based approach to postsurgical, at-home exercises can be beneficial, like after a thoracotomy. VR interfaces have already been used successfully as an adjunct to existing post–cardiac surgery rehabilitation protocols like physical therapy. The same study results showed beneficial outcomes for the VR-treated group, including less postoperative pain, better functional performance, better walking capacity, and higher energy levels.

There are also many potential strategic benefits of incorporating XR into cardiovascular programs. XR also has the potential to save costs by reducing procedure times and the number of people required in the procedure room. In addition, physicians may be attracted to CV programs that incorporate XR. Being able to see the heart and its interactions with surrounding organs in a new, more concrete fashion may boost physician confidence. And medical device companies with XR innovations may be looking for cardiovascular programs to champion their technology and bring it to market, which could result in the formation of mutually beneficial partnerships.

Extended reality isn’t just the stuff of the video game and entertainment industries. One survey from VR Intelligence found that 65% of surveyed AR companies in 2020 were working on industrial applications only, and not consumer products and software. Additionally, Business Wire estimates the global market for AR and VR in health care is expected to reach $10.8 billion by 2025, representing a CAGR of 36% from 2019 through 2026.

The confection of extended reality technologies and their applicability to cardiovascular surgery is undergoing a transformation, and many people believe it’s better to embrace it sooner rather than later. Multiple cardiovascular programs across the United States have already acquired this technology and implemented it for surgeon training, care planning, and surgery execution. These moves have been driven by technology innovation and maturation, upticks in research studies, investments from corporate players, and increased dialogue on XR's benefits and applications across the health care industry. These developments represent only the beginning of what this technology may be able to provide the industry across the next decade.

Lucile Packard Children’s Hospital at Stanford

Lucile Packard Children’s Hospital (LPCH) at Stanford is innovating physician and patient education on congenital heart defects by integrating extended reality into training through the Stanford Virtual Heart program.

LPCH supports a large patient population with rare congenital heart defects which often result in complex surgeries that are difficult to visualize and explain to physicians, patients, and families. As a result, David Axelrod, an attending physician at LPCH Stanford and clinical professor of pediatric cardiology, began exploring ways to better visualize and explain these defects. In 2016, he partnered with Lighthaus, an immersive learning technology company, to build XR education platforms for congenital heart defects, which became known as the Stanford Virtual Heart.

Stanford Virtual Heart has shown that XR can enhance physician education on complex conditions that can be difficult to visualize with 2D imaging. Dr. Axelrod is currently conducting research on the impact of XR on physician education about congenital heart defects, and early findings indicate that VR is either equivalent to or better than standard education for anatomical questions. Additionally, students prefer VR training as it provides a better visual and spatial understanding of complex anatomies.

XR can also help patients and families understand congenital heart defects in new ways. After reviewing 3D modeling of her child’s heart defect, one mother said that she was finally able to understand a heart defect that had been in her family for 15 years.

Physician education is the first use case for XR in cardiovascular surgery at LPCH Stanford, but the organization plans to pursue preoperative applications in the future. LPCH is also interested in the future use of simulations which “age” virtual surgeries to see how the heart will function millions of heart beats after surgery.

CentraCare

CentraCare is using XR to reduce Watchman procedure times and improve outcomes through 3D and 4D pre-planning and intra-operative technology.

CentraCare Heart and Vascular Center is one of the largest Cardiovascular programs in Minnesota and is ranked #2 in the state and #35 in the nation for Cardiology and Heart Surgery according to the most recent ranking by USNews Reports. The center oversees a high volume of interventional and structural heart procedures including left atrial appendage occlusion (LAAO) using the Watchman and Amulet devices. These procedures are technically complex which can be challenging to plan for and perform with standard 2D imaging. Jacob Dutcher, Interventional Cardiologist and Director of the Structural Heart Program at CentraCare, recognized this challenge and sought a solution aided by technology. He connected with EchoPixel, a technology start-up that created a tool which converts standard 2D CT and ultrasound images into 3D and 4D holograms, respectively. This advanced imaging is a new way to visualize and understand the anatomy of the heart and has dramatically changed the way interventionalists approach the LAAO procedure.

Dr. Dutcher worked with EchoPixel to integrate this technology, and the 3D pre-procedural planning software helped CentraCare Heart and Vascular Center reduce LAAO procedures times by more that 27% and increase optimal procedure outcomes by 20%, as presented at the international American College of Cardiology conference in March 2020. Dr. Dutcher’s program has completed some of the fastest LAAO procedures in the world with procedure times frequently less than 20 minutes and occasionally in as little as 10 minutes. This has led to their ability to performed the procedure routinely without conscious sedation and discharge patients home in as little as 5 hours after completion of the procedure.

Following this success, CentraCare combined EchoPixel’s pre-procedural planning 3D software with its intra-operative 4D Holographic Therapy Guidance (HTG) software platform. The new 4D HTG software provides an innovative 4D experience that allows the heart team to interact with a patient’s specific organs as if they were physical objects. In May of 2021 Dr. Dutcher performed the world’s first structural heart procedure using 4D hologram technology.

They have now utilized this technology for approximately 40 LAAO procedures and have found that 4D hologram technology has provided them with more confidence in device placement while reducing the amount of radiation and contrast needed to perform the procedure. Dr. Dutcher also now routinely used conscious sedation when the EchoPixel 4D hologram is used. This reduces the cost of the procedure and expedites procedure room turn-over time, increasing the efficiency of the lab and allowing for higher daily volume of procedures to be performed. He also believes the integration of 3D and 4D hologram technology will ultimately help their structural heart program reduce the number of staff needed in the procedure room, further reduce procedure time, improve patient safety and outcomes, and vastly increase physician confidence in device placement.

The integration and application of extended reality in cardiovascular surgery is still nascent. Though most organizations will want to wait to see it develop further, those who might consider early adoption of XR include those with:

• Executive buy-in and organizational goals that align with the goals for acquiring the technology
• A comprehensive cardiovascular program, institute, or center
• Teaching and research capabilities
• An appropriate case mix of structural heart procedures
• An appropriate physician mix of surgeons performing structural heart procedures or conducting/completing training, as well as a physician to champion the technology implementation
• Access to capital to allot to this investment or in partnership with a manufacturer

Strategists at organizations meeting these criteria should speak with their cardiovascular leaders about pursuing this technology.

Pros

• May result in better patient outcomes via more precise surgeries
• May carry a cost-saving potential via fewer people in the procedure room, shorter procedure times, and fewer complications
• May make an organization be seen as innovative and progressive in the market

Cons

• May be prohibitively expensive due to procurement cost
• Training staff can create usability challenges
• Gaining executive buy-in can slow the path to procurement or halt it completely

At times it feels as if we have only discovered the tip of the iceberg of ways extended reality can be embedded in health care. While the proliferation of XR in cardiovascular surgery has so far been low, due to limited provider and therefore patient access to it and high costs of procurement, there are several factors that could help increase its adoption soon.

Things that change the calculus:

• More real-world evidence of the superiority of XR modalities in physician training, as well as planning and performing surgery
• Expanded provider access to the technology
• Minimal additional costs to patients

In terms of longer-term applicability, procedures like the Watchman are just the beginning of applications of 3D and 4D hologram technology for structural heart procedures. Some providers hope to expand to mitral-based procedures, plugs for perivalvular leaks, conscious sedations without a probe, and EP for complex ablations. The potential usability of XR and positive patient outcomes compared to traditional methods and techniques will be on our radar for the future.