Project I4: Shared Control in Robotic, Minimally Invasive Surgery

In recent years, minimally invasive surgery has drawn the attention of both surgeons and researchers on the field of robotics. While the former developed new techniques for surgical interventions in order to improve the treatment of patients, the latter have predominantly worked on the enhancement of the applicability of minimally invasive instruments. Therefore, as part of project I4, a novel robotic system for minimally invasive surgery was developed at the department. The instruments of the system can be controlled remotely by a surgeon sitting at a master console, which can be placed somewhere in the operation theater. The master console is equipped with sophisticated input devices, which provide an intuitive handling of the instruments (Cartesian control without any chopstick effect). Usually, these advantages come at the price of reduced immersiveness, since the surgeon cannot feel any forces exerted onto the situs. In addition, working speed is significantly reduced, which renders these types of operations stressful for both patients and surgeons. Therefore the main focus of research is on incorporating force feedback and automation into the system. We have chosen robotic knot-tying as a reference application since it combines many of the difficulties occurring in automation. One of them is the handling of limp objects like surgical threads, whose positional behavior can hardly be predicted.

Video

learned_knot.wmv: film of an automatically applied surgical knot

Open Student Projects

Project T8: Miniaturized image sensors for medical instruments and applications

The miniaturization of image sensors in medical applications enables new and inovative applications. Together with our industry parter, a manufacturer of ultra-small, customized CMOS sensors, this project aims to equip minimally invasice surgical instrument with additional cameras. In combination with the existing stereoscopic endoscope, usually available in telepresent surgical scenarios, the redundant image information can now dramatically improve two main limitations, navigation and orientation of the surgeon. Multiple viewing angles might improve 3D scene recognition and increase the viewing angle. As the camera is directly attatched to the surgical instrument, also automated tasks such as grasping benefit from the gained precission.

Project T4: Robot-Assisted Handling of Limp and Deformable Objects

Like all other transfer projects, T4 is conducted in cooperation with an industrial partner. We were able to acquire Mitsubishi Electric as a partner for this project, a company with outstanding experience on the field of industrial robotics. The primary objective of this project is the transfer and application of expertise, which was obtained in project I4: i.e. techniques for the recognition, tracking, modelling and prediction of deformable objects (particularly thread material) in surgical environments will be transferred to industrial applications. Analogies are manifold: surgical threads show high felxibility, deformability and limpness. These features can also be found in assembly material like cables, wire straps, tubes and sealings. Due to the mentioned material properties, handling of such structures is extremely diffcult.

more...

People

Dr. Eva U. Braun (Medical Evaluation)

Former Members

Dr. Hermann Mayer (Robotics, Automation)
Dr. Christoph Staub (Robotics, Automation)
Dr. Thomas Müller (Robotics, Automation)
Franz Murr (Technical Assistant)
Dipl.-Inf. Istvan Nagy (Stereo-Vision, 3D Reconstruction)

Master Students

Andreas Carstens: Generierung von Kontrollsequenzen für die Roboterchirurgie
Timo Gebhart: Automatisches Knotenlegen in der minimal-invasiven Herzchirurgie
Stefan Hueber: Erweiterung einer Simulationsumgebung zur Robotersteuerung
Patrick Maier: Robotersteuerung per PDA über WLAN / Bluetooth
Michael Stegherr: Entwurf einer 3D Benutzeroberfläche für die Roboterchirurgie
Boris Vacsic: Ansteuerung minimalinvasiver Instrumente über RS232 / CAN-Bus

Acknowledgement

This project is part of the Collaborative Research Centre 453 (SFB 453: TWirklichkeitsnahe Telepräsenz und Teleaktion), which is funded by the German Research Foundation (DFG). The research is also supported by the German Heart Center Munich, which has provided its medical facilities to install and evaluate the robotic system.

Selected Publications

[1]	Thomas Müller, Binh An Tran, and Alois Knoll. Automatic distribution of vision-tasks on computing clusters. In Proceedings of IS&T / SPIE Electronic Imaging, 2011. [ DOI \| .bib \| .pdf ]
[2]	Thomas Müller and Alois Knoll. Evolution of attention mechanisms for early visual processing. In Proceedings of IS&T / SPIE Electronic Imaging, 2011. [ DOI \| .bib \| .pdf ]
[3]	Nguyen Xuan Tuong, Thomas Müller, and Alois Knoll. Robust pedestrian detection and tracking from a moving vehicle. In Proceedings of IS&T / SPIE Electronic Imaging, 2011. [ DOI \| .bib \| .pdf ]
[4]	Thomas Müller and Alois Knoll. A generic approach to realtime robot control and parallel processing for industrial scenarios. In Proceedings of the IEEE International Conference on Industrial Technology, 2010. [ DOI \| .bib \| .pdf ]
[5]	Thomas Müller, Binh An Tran, and Alois Knoll. A flexible robotics and automation system: Parallel visual processing, realtime actuator control, and task automation for limp object handling. In Proceedings of the 7th International Conference on Informatics in Control, Automation and Robotics, 2010. [ .bib \| .pdf ]
[6]	Marwan Radi, Andrea Reiter, Michael F. Zäh, Thomas Müller, and Alois Knoll. Telepresence technology for production: From manual to automated assembly. In Proceedings of EuroHaptics 2010, 2010. [ DOI \| .bib \| .pdf ]
[7]	Thomas Müller and Alois Knoll. Virtualization techniques for cross platform automated software builds, tests and deployment. In Proceedings of the Fourth International Conference on Software Engineering Advances, Porto, Portugal, 2009. [ .bib \| .pdf ]
[8]	Hermann Mayer, Darius Burschka, Alois Knoll, Eva U. Braun, Rüdiger Lange, and Robert Bauernschmitt. Human-machine skill transfer extended by a scaffolding framework. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 2866-2871, 2008. [ DOI \| .bib \| .pdf ]
[9]	Hermann Mayer. Human Machine Skill Transfer in Robot Assisted, Minimally Invasive Surgery. PhD thesis, Technische Universität München, 2008. [ .bib \| .pdf ]
[10]	Thomas Müller and Alois Knoll. Humanoid early visual processing using attention mechanisms. In Proceedings of Cognitive Humanoid Vision WS at IEEE-RAS International Conference on Humanoid Robots, Daejeon, Korea, 2008. [ .bib \| .pdf ]
[11]	Hermann Mayer, Istvan Nagy, Alois Knoll, Eva U. Braun, and Robert Bauernschmitt. Adaptive control for human-robot skill transfer: Trajectory planning based on fluid dynamics. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 1800-1807, 2007. [ DOI \| .bib \| .pdf ]
[12]	Hermann Mayer, Istvan Nagy, Alois Knoll, Eva U. Braun, Robert Bauernschmitt, and Rüdiger Lange. Haptic feedback in a telepresence system for endoscopic heart surgery. MIT Presence: Teleoperators and Virtual Environments, 16(5):459-470, 2007. [ DOI \| .bib \| .pdf ]
[13]	Hermann Mayer, Faustino Gomez, Daan Wierstra, Istvan Nagy, Alois Knoll, and Jürgen Schmidhuber. A system for robotic heart surgery that learns to tie knots using recurrent neural networks. Advanced Robotics. Special Issue: Selected Papers from IROS 2006, 21(14):1521-1537, 2007. [ DOI \| .bib \| .pdf ]
[14]	Hermann Mayer, Faustino Gomez, Daan Wierstra, Istvan Nagy, Alois Knoll, and Jürgen Schmidhuber. A system for robotic heart surgery that learns to tie knots using recurrent neural networks. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 543-548, Beijing, China, 2006. [ DOI \| .bib \| .pdf ]
[15]	Eva U. Schirmbeck, Constanze Haßelbeck, Hermann Mayer, Istvan Nagy, Alois Knoll, Franziska Freyberger, Michael Popp, Rüdiger Lange, and Robert Bauernschmitt. Evaluation of haptics in robotic heart surgery. International Journal of Computer Assisted Radiology and Surgery, 1(1):730-734, 2005. [ DOI \| .bib \| .pdf ]
[16]	Hermann Mayer, Istvan Nagy, Alois Knoll, Eva U. Schirmbeck, and Robert Bauernschmitt. The Endo[PA]R system for minimally invasive robotic surgery. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3637-3642, Sendai, Japan, 2004. [ .bib \| .pdf ]
[17]	Istvan Nagy, Hermann Mayer, and Alois Knoll. Application of force feedback in robot assisted minimally invasive surgery. In Proceedings of the 4th International Conference Eurohaptics, pages 240-245, Munich, Germany, 2004. [ DOI \| .bib \| .pdf ]
[18]	Hermann Mayer, Istvan Nagy, and Alois Knoll. Skill transfer and learning by demonstration in a realistic scenario of laparoscopic surgery. In Proceedings of the IEEE International Conference on Humanoids CD-ROM, Munich, Germany, 2003. [ .bib \| .pdf ]