Surgical Robotics R&D
Developing intraoperative guidance overlays for use in robotic surgery.
Problem
During procedures such as robot-assisted laparoscopic prostatectomy (RALP), surgeons must navigate dynamic, patient-specific anatomy where critical structures, including nerves and vessels, may be obscured from view. By combining preoperative and intraoperative imaging, this effort aims to provide real-time 3D guidance directly within the robotic console, helping surgeons navigate sensitive anatomy more precisely and reduce morbidity during tumor removal.
Contribution
As demonstrated in the video below, I programmed a system for overlaying registered 3D geometry onto its real-world counterpart within a da Vinci surgical console. Using the lab's NDI Vega optical tracker and a workstation equipped with a Bluefish video card, I built the registration, tracking, rendering, and real-time video overlay pipeline needed to bring virtual anatomy into the live stereoscopic feed and display it to the surgeon console. The system can integrate across platforms including the da Vinci S, SP, and Xi. The software has since been substantially expanded by the team into MIRAGE1, while retaining frameworks I first implemented, including the stereo OpenGL rendering pipeline and ImGui-based interface.
Evaluation
Registration uncertainty, optical tracking error, and camera extrinsic and intrinsic calibration could each contribute to mismatch between the rendered overlay and the physical scene. To quantify this error, I developed a validation tool that projected a registered virtual checkerboard onto a physical checkerboard in the endoscopic video. Because corresponding intersections should coincide, their displacement provided a direct pixel-space error measurement. Comparing da Vinci platforms, I found substantially better camera calibration performance on the newer Xi than on the S. The team later decided to apply the same metric to compare endoscope-tracking approaches, which was published in this SPIE paper I coauthored.
I'm grateful to the entire Task 5 team, including Dr. Halter, Dr. Nasiri, Dr. Zheng, Dr. Fan, and Dr. Kokko, for their guidance, collaboration, and support throughout the spring.