DREAMING 2024¶
Diminished Reality for Emerging Applications in Medicine through Inpainting¶
Part of ISBI 2024 in Athens, Greece!
Challenge concluded!¶
We want to thank all participants for their involvement in the IEEE ISBI DREAMING challenge. We are delighted that our first venture into DR in a medical setting was well received, and we are pleased with the number of submissions we received!
As many of you noticed, there were some unexpected extensions to the method submission deadline. These were due to unforeseen resource limitations on the grand-challenge platform and several account issues. We understand that this caused some irritation among participants and apologize for any inconvenience this caused. We appreciate your patience and understanding and are exploring ways to fairly recognize the best-performing method under the circumstances.
We hope this challenge has sparked interest in the field of medical DR and that the field continues to grow, and are looking forward to meet some of you at ISBI 2024 in Athens! If you have any questions or feedback, we would be happy to hear from you!
Best regards,
The DREAMING Challenge Organization Team
Description¶
The Diminished Reality for Emerging Applications in Medicine through Inpainting (DREAMING) challenge seeks to pioneer the integration of Diminished Reality (DR) into oral and maxillofacial surgery. While Augmented Reality (AR) has been extensively explored in medicine, DR remains largely uncharted territory.
DR involves virtually removing real objects from the environment by replacing them with their background. Recent inpainting methods present an opportunity for real-time DR applications without scene knowledge. DREAMING focuses on implementing such methods to fill obscured regions in surgery scenes with realistic backgrounds, emphasizing the complex facial anatomy and patient diversity.
The challenge provides a dataset of synthetic yet photorealistic surgery scenes featuring humans, simulating an operating room setting. Participants are tasked with developing algorithms that seamlessly remove disruptions caused by medical instruments and hands, offering surgeons an unimpeded view of the operative site.
Background¶
While Augmented Reality (AR) is extensively studied in medicine, it represents just one possibility for modifying the real environment. Other forms of Mediated Reality (MR) remain largely unexplored in the medical domain. Diminished Reality (DR) is such a modality. DR refers to the removal of real objects from the environment by virtually replacing them with their background [1]. Combined with AR, powerful MR environments can be created. Although of interest within the broader computer vision and graphics community, DR is not yet widely adopted in medicine [2]. However, DR holds huge potential in medical applications. For example, where constraints on space and intra-operative visibility exist, and the surgeons' view of the patient is further obstructed by disruptive medical instruments or personnel [3], DR methods can provide the surgeon with an unobstructed view of the operation site. Recently, advancements in deep learning have paved the way for real-time DR applications, offering impressive imaging quality without the need for prior knowledge about the current scene [4]. Specifically, deep inpainting methods stand out as the most promising direction for DR [5,6,7].
Task¶
The DREAMING challenge focuses on implementing inpainting-based DR methods in oral and maxillofacial surgery. Algorithms shall fill regions of interest concealed by disruptive objects with a plausible background, such as the patient's face and its surroundings. The facial region is particularly interesting for medical DR, due to its complex anatomy and variety through age, gender and ethnicity. Therefore, we provide a dataset consisting of synthetic, but photorealistic, surgery scenes focusing on patient faces, with obstructions from medical instruments and hands holding them. These scenes are generated by rendering highly realistic humans together with 3D-scanned medical instruments in a simulated operating room (OR) setting.
This challenge represents an initial frontier in the realm of medical DR, offering a simplified setting to pave the way for MR in medicine. In the future, the potential for more sophisticated applications is expected to unfold.
References¶
1. Mori, S., Ikeda, S., & Saito, H. (2017). A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects. IPSJ Transactions on Computer Vision and Applications, 9(1), 1-14. Article
2. Ienaga, N., Bork, F., Meerits, S., Mori, S., Fallavollita, P., Navab, N., & Saito, H. (2016, September). First deployment of diminished reality for anatomy education. In ISMAR-Adjunct (pp. 294-296). IEEE. Article
3. Egger, J., & Chen, X. (Eds.). (2021). Computer-Aided Oral and Maxillofacial Surgery: Developments, Applications, and Future Perspectives. Academic Press. Book
4. Gsaxner, C., Mori, S., Schmalstieg, D., Egger, J., Paar, G., Bailer, W. & Kalkofen, D. (2023). DeepDR: Deep Structure-Aware RGB-D Inpainting for Diminished Reality. arXiv preprint arXiv: 2312.00532. Preprint
5. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016). Context encoders: Feature learning by inpainting. In CVPR (pp. 2536-2544). Article
6. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2019). Free-form image inpainting with gated convolution. In ICCV (pp. 4471-4480). Article
7. Kim, D., Woo, S., Lee, J. Y., & Kweon, I. S. (2019). Deep video inpainting. In CVPR (pp. 5792-5801). Article