Abstract
Automatic recognition of surgical phases is an important component for developing an intra-operative context-aware system. Prior work in this area focuses on recognizing short-term tool usage patterns within surgical phases. However, the difference between intra- and inter-phase tool usage patterns has not been investigated for automatic phase recognition. We developed a Recurrent Neural Network (RNN), in particular a state-preserving Long Short Term Memory (LSTM) architecture to utilize the long-term evolution of tool usage within complete surgical procedures. For fully automatic tool presence detection from surgical video frames, a Convolutional Neural Network (CNN) based architecture namely ZIBNet is employed. Our proposed approach outperformed EndoNet by 8.1% on overall precision for phase detection tasks and 12.5% on meanAP for tool recognition tasks.
Introduction
Decomposing a surgical process into a sequence of abstract tasks independent of the physical factors was introduced by MacKenzie et al. [1] under the term Surgical Process Modeling (SPM). SPM centers around a concept of multi-scale temporal abstraction – termed as granularity [2]. In SPM, the highest granularity level is called surgical phase. A fully automatic recognition of such phases can enable multiple applications [3], [4]. Understanding the temporal evolution of tool usage patterns and discovering their relation to respective surgical phases can provide important cues for an automatic recognition of surgical phases [5]. Henceforth, by the term evolution, we will refer to the sequential process of surgical tool usage (including co-occurrence) over a complete surgical procedure. The term pattern refers to the visibility of tools within endoscopic images.
The general trend in literature on fully automatic surgical phase recognition is to explicitly model a low level SPM abstraction followed by a global temporal alignment, using some variant of Hidden Markov Models (HMM). For example, Padoy et al. [5] and Twinanda et al. [6] modeled tool presence information using Dynamic Time Warping (DTW) and CNN respectively, followed by a hierarchical HMM. Dergachyova et al. [7] used image and instrument signals for modeling low level information, followed by a Hidden semi Markov Model for global alignment. A spatio-temporal CNN was introduced by Lea et al. [8] for modeling tool motion followed by a semi Markov model or DTW. LSTM has been introduced by DiPietro et al. [9] for surgical phase recognition from kinematics data and more recently utilized by Jin et al. [10] in combination with low-level image features (obtained from a CNN) and heuristic post processing.
Key observations of the present work are the inter- and intra-phase differences in tool usage evolution, as shown in Figure 1. Unlike prior methods, which solely learn within-phase information, we learn key changes for sequences of tool usage that uniquely identify phase transitions in an endoscopic video sequence. In its essence, the proposed method utilizes tool presence prediction of ZIBNet [11] in a state-preserving LSTM [12] framework for encoding complete evolution of tool usage during a surgical procedure.
Corrplot visualization of the evolution of surgical tool usage (including tool co-occurrences) at different phases of a surgery as provided with the Cholec80 dataset.
The major contribution of the proposed work is a novel framework for learning evolution of surgical tool usage patterns in relation to inter- and intra-phases, where many-to-one LSTM sequence learning is utilized for fully automatic learning of long-term tool usage evolution.
Method
A set of video sequences, each composed of individual frames
Overall design of the proposed surgical phase recognition framework.
ZIBNet
Sahu et al. introduced ZIBNet as a specialized transfer learning framework developed on top of generic CNN feature learning architectures for surgical tool presence detection [13], [11]. ZIBNet considers surgical tool presence detection as a multi-label classification task and tool co-occurrences (second and third order) are treated as separate classes. A detrimental effect of imbalance in tool usage on the performance of the CNN was analyzed. A stratification technique to counter the imbalance was employed during CNN training. Moreover, online post-processing using temporal smoothing was introduced to enhance run time prediction. In contrast to [11] which only considered AlexNet [14] as the base CNN architecture, we also investigate Residual Neural Networks (ResNet [15]) trained for ImageNet classification as the base CNN architecture of ZIBNet.
Considering a number of L tools are necessary to perform a surgical procedure, ZIBNet learns the mapping between
State-preserving LSTM
The goal of an LSTM is to learn the mapping between tool labels
Typical LSTM architectures map an input with fixed dimensionality to an output vector. However, this setup offers a serious limitation for online phase detection since the length of a surgical procedure is not known a-priori. One key design choice of our particular learning strategy is to focus on learning phase transitions rather than the overall description of phases. In this work, we formulate the online phase detection as a many-to-one LSTM sequence learning task. Figure 2 visualizes the pipeline of the proposed method. In particular, the input sequence of tool predictions is fed into the LSTM, one mini-sequence at a time, and the respective state of the LSTM is preserved for the following mini-sequence in order to retain long term dependency.
Experiments and results
The performance of the proposed method was quantitatively evaluated in view of a fully automatic surgical phase recognition. In the following, the used data set, the experimental strategy, the quantitative analysis and the comparison with related state-of-the-art methods are described.
Data preparation and experimental settings
All our experiments were conducted with the Cholec80 dataset [6] in an online setting (i.e. no future information was used for training). The Cholec80 dataset comprises 80 videos of cholecystectomy procedures performed by 13 surgeons. The frame rate of 25 fps was down-sampled to 1 fps for a ground truth annotation of tool labels as well as for further processing. For SPM, a cholecystectomy procedure is divided into seven surgical phases, where seven tools are commonly used to perform the procedure (see Figure 1). We follow the evaluation strategy of Twinanda et al. [6] for both tool presence detection and phase recognition in order to provide direct comparison with their method.
The training data was created by converting the video into mini-sequences of five frames while maintaining the original sequential order. We utilized eight LSTM units, categorical cross entropy as loss function and Adam as optimization algorithm with a learning rate of 0.001.
Quantitative comparison
Our proposed framework outperforms EndoNet [6] on phase recognition evaluation metrics as developed by Padoy et al. [5], namely average precision and average recall. Results of top performing methods on the Cholec80 dataset from Twinanda et al. [6] are listed together with our results in Table 1. In particular, the performance of our proposed method is compared to those of PhaseNet, EndoNet and EndoNet followed by HHMM (EndoNet + HHMM) based global alignment. Our proposed method leads to approx. 8% improvement in terms of average precision, compared to the second best result (EndoNet + HHMM).
ZIBNet performance for tool detection
Due to the dependence of our proposed method on tool presence, it is important to employ an accurate and reliable method for the detection of tool presence right at the outset. To this end, we compared the performance of two state-of-the-art tool detection methods on the Cholec80 dataset using Average Precision (AP) as a metric. We considered two flavors of ZIBNet [11] with base CNN as either AlexNet [14] or ResNet [15], reported as ZIBNet-AlexNet and ZIBNet-ResNet respectively. It becomes evident from Table 2 that both versions of ZIBNet outperformed EndoNet [6] in terms of Mean Average Precision (MeanAP) for all tools. However, the ResNet flavor beats the two other methods in each tool category and achieved an overall MeanAP of 93.5. An interesting observation is that the less frequent tools like scissors, irrigators, specimen bags etc. are more related to phase transitions. The general superior performance of ZIBNet over EndoNet for these tools (for example, an increase by approx. 30% for scissors), as shown in Table 2, prompted us to choose ZIBNet with ResNet [15] for initial fully automatic tool presence detection. Note that in Tables 1 and 3 the performance of ResNet flavor of ZIBNet is reported in the columns entitled Proposed.
Comparison of tool recognition performance between EndoNet [6] (EN) and two flavors of ZIBNet: ZIBNet-AlexNet (ZA) and ZIBNet-ResNet (ZR) on the Cholec80 dataset.
| Tools | EN | ZA | ZR |
|---|---|---|---|
| Grasper | 84.8 | 86.6 | 90.5 |
| Bipolar | 86.9 | 94.8 | 95.6 |
| Hook | 95.6 | 98.8 | 99.3 |
| Scissors | 58.6 | 68.8 | 86.8 |
| Clipper | 80.1 | 85.0 | 96.5 |
| Irrigator | 74.4 | 77.2 | 89.7 |
| Specimen bag | 86.8 | 88.3 | 96.1 |
| MeanAP | 81.0 | 85.7 | 93.5 |
Analysis and comparison of recognition performance for each phase using Precision(Recall) on the Cholec80 dataset (EH → EndoNet + HHMM [6])
| Phases | EH | Proposed |
|---|---|---|
| Preparation | 90.0 (85.5) | 99.1 (87.6) |
| Calot triangle dissection | 96.4 (81.1) | 98.5 (85.3) |
| Clipping and cutting | 69.8 (71.2) | 73.6 (68.8) |
| Gall bladder dissection | 82.8 (86.5) | 81.6 (92.7) |
| Gall bladder packaging | 55.5 (75.5) | 83.0 (83.4) |
| Cleaning and coagulation | 63.9 (68.7) | 64.7 (77.4) |
| Gall bladder retraction | 57.5 (88.9) | 71.9 (68.1) |
| Average | 73.7 (79.6) | 81.8 (80.9) |
Quantitative analysis
We quantitatively analyzed the specific design choices of our proposed method and its performance on each of the seven surgical phases. Precision and recall are considered as the metric for measuring the performance of fully automatic phase recognition. We specified the combined performance of EndoNet and HHMM (reported as EndoNet + HHMM) for each surgical phase in Table 3. Our proposed method outperformed EndoNet in terms of precision and recall.
Our proposed framework is implemented in Keras-Theano, running at 6 fps on an NVIDIA GTX 1080 Ti when applied on a surgical video.
Discussions and conclusion
We presented a fully automatic technique for the recognition of surgical phases from endoscopic videos of cholecystectomy procedures. Unlike prior work, we focused on differentiating between evolution of tool usage at phase transitions and within phases. In particular, a many-to-one state-preserving LSTM was trained on the tool presence predictions of ZIBNet to learn evolution of tool transition patterns from endoscopic videos. The proposed method outperformed EndoNet methods on the Cholec80 dataset by 8.1% on average precision. This study solely concentrated on tool presence as the initial source of information. However, future studies of surgical phase recognition from endoscopic videos may benefit from studying tool localization on the image context as the initial source. Finally, such fully automatic techniques are expected to be instrumental in advancing the computer assistance during surgical intervention from bench to bedside.
Funding source: German Federal Ministry of Education and Research (BMBF)
Award Identifier / Grant number: 16 SV 8019
Research funding: This study was funded by the German Federal Ministry of Education and Research (BMBF) under the project COMPASS (grant no. - 16 SV 8019).
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
Conflict of interest: The authors state no conflict of interest.
Informed consent: This study contains patient data from a publicly available dataset.
Ethical approval: This article does not contain any studies with human participants or animals performed by any of the authors.
References
1. MacKenzie, L, Ibbotson, J, Cao, C, Lomax, A. Hierarchical decomposition of laparoscopic surgery: a human factors approach to investigating the operating room environment. Minim Invasive Ther Allied Technol 2001;10:121–7. https://doi.org/10.1080/136457001753192222.Search in Google Scholar
2. Lalys, F, Jannin, P. Surgical process modelling: a review. IJCARS 2014;9:495–511. https://doi.org/10.1007/s11548-013-0940-5.Search in Google Scholar
3. Blum, T, Feußner, H, Navab, N. Modeling and segmentation of surgical workflow from laparoscopic video. In: MICCAI Springer; 2010, pp. 400–7.10.1007/978-3-642-15711-0_50Search in Google Scholar PubMed
4. Franke, S, Meixensberger, J, Neumuth, T. Multi-perspective workflow modeling for online surgical situation models. J Biomed Inf 2015;54:158–66. https://doi.org/10.1016/j.jbi.2015.02.005.Search in Google Scholar
5. Padoy, N, Blum, T, Ahmadi, SA, Feussner, H, Berger, MO, Navab, N. Statistical modeling and recognition of surgical workflow. Med Image Anal 2012;16:632–41. https://doi.org/10.1016/j.media.2010.10.001.Search in Google Scholar
6. Twinanda, AP, Shehata, S, Mutter, D, Marescaux, J, de Mathelin, M, Padoy, N. EndoNet: a deep architecture for recognition tasks on laparoscopic videos. In: IEEE TMI 2017;36:86–97. https://doi.org/10.1109/TMI.2016.2593957.10.1109/TMI.2016.2593957Search in Google Scholar PubMed
7. Dergachyova, O, Bouget, D, Huaulmé, A, Morandi, X, Jannin, P. Automatic data-driven real-time segmentation and recognition of surgical workflow. IJCARS 2016;11:1081–9. https://doi.org/10.1007/s11548-016-1371-x.Search in Google Scholar
8. Lea, C, Choi, JH, Reiter, A, Hager, GD. Surgical phase recognition: from instrumented ORs to hospitals around the world. M2CAI workshop, MICCAI; 2016.Search in Google Scholar
9. DiPietro, R, Lea, C, Malpani, A, Ahmidi, N, Vedula, SS, Lee, GI, et al. Recognizing surgical activities with recurrent neural networks. In: MICCAI, Springer; 2016, pp. 551–8.10.1007/978-3-319-46720-7_64Search in Google Scholar
10. Jin, Y, Dou, Q, Chen, H, Yu, L, Heng, PA. SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network. In: IEEE TMI 2018;37:1114–26. https://doi.org/10.1109/TMI.2017.2787657.Search in Google Scholar
11. Sahu, M, Mukhopadhyay, A, Szengel, A, Zachow, S. Addressing multi-label imbalance problem of surgical tool detection using CNN. IJCARS 2017;12:1013–20. https://doi.org/10.1007/s11548-017-1565-x.Search in Google Scholar
12. Hochreiter, S, Schmidhuber, J. Long short-term memory. Neural Comput 1997;9:1735–80. https://doi.org/10.1162/neco.1997.9.8.1735.Search in Google Scholar
13. Sahu, M, Mukhopadhyay, A, Szengel, A, Zachow, S. Tool and phase recognition using contextual CNN features. Tech report – M2CAI challenge. MICCAI; 2016.Search in Google Scholar
14. Krizhevsky, A, Sutskever, I, Hinton, GE. Imagenet classification with deep convolutional neural networks. In: NeurIPS; 2012, pp. 1097–105.10.1145/3065386Search in Google Scholar
15. He, K, Zhang, X, Ren, S, Sun, J. Deep residual learning for image recognition. In: IEEE CVPR; 2016, pp. 770–8.10.1109/CVPR.2016.90Search in Google Scholar
© 2020 Manish Sahu et al., published by De Gruyter, Berlin/Boston
This work is licensed under the Creative Commons Attribution 4.0 International License.
