Developing and Assessing

Virtual Annotation for Robotic Surgery

A Unity-based 3D annotation environment for helping surgeons and pathologists markup 3D-scanned specimens across platforms.

  • Adapted into publication submission
  • Highlighted in an NIH grant report
  • Generated an intellectual property disclosure
Residents testing my annotation environment with a da Vinci S surgical system.

Clinical Target

Margins Are Spatial

Following transoral robotic surgery (TORS), communication between surgeons and pathologists regarding complex resected tissue can be imprecise and often still relies on pencil and paper. My goal was to design, implement, and evaluate an adaptable 3D annotation environment capable of integrating with surgical robots (e.g. left), VR headsets, and typical computers. After quickly demonstrating functionality, iterative improvements were guided by my advisor Dr. Kokko as well as physicians like Dr. Joseph Paydarfar MD.

TORS specimen communication diagram

User Testing

Study & Results

To gather quantitative and qualitative feedback, I designed and conducted a short study. Participants included attending physicians, a pathologist, residents, medical students, and college undergraduates. One task involved color 3D-printing a real specimen replica, as reconstructed by Dr. Kokko from endoscopic video1. Marking it up digitally, I asked participants to recreate the annotations virtually in my program to test.

|

The results of this study, and more details on the program implementation, will be published by the College in my full thesis. In addition, the following paper has been submitted to SPIE Medical Imaging 2027:

Thesis title page preview
TORS specimen communication diagram
Physical goal I handed participants (left), and examples of their virtual recreations in my program (right).

Thank You

I'm very grateful to my lab mentor Michael Kokko for his support on this project, as well as Prof. Shepherd, Dr. Paydarfar, Ryan Halter, Michelle Yoon, and others. This contributed to a wider TORS effort at Dartmouth, supported by an NIH R21 grant. URAD and the Neukom Institute supported my earlier work leading into the project.