Abstract
In this study we evaluated open-source software for lung segmentation. Several parameters which emphasize on functionality, usability, image quality and 3D export are considered for this evaluation. Based on these parameters, a scoring system is generated. Our preliminary evaluation results indicated that the Pulmonary toolkit obtains the best overall performance according to the scoring system. However, the ranking of software shows a certain variation among different criteria. The selection of software should regard the focus and the specific interest of the user.
1 Introduction
Segmentation of the lungs from Computer Tomography (CT) images is considered to be a fundamental vital step in the field of medical imaging and pulmonary image analysis [1]. The need for separating lungs from surrounding tissue, as well as generating 3D models makes it necessary to develop software and toolkits that can perform the segmentation robustly and accurately with minimal user intervention.
In this paper, open-source software for the semi-automatic segmentation of the lungs are compared in terms of their functionality, usability, image quality of segmentation and file or data types of 3D volumes that can be exported. The results obtained are used to evaluate the software, and to deduce which software has better performance for the segmentation and 3D model generation of the lungs.
2 Methods
2.1 Selection of software and toolkits
Seven software and toolkits for the segmentation of the lungs and the airways have been found through Internet:
The Pulmonary toolkit/Github (Ptk for short)*
YaDiv
NIH-CIDI Lung Segmentation tool
TurtleSeg
MITK*
ITK-SNAP*
3D Slicer*
*software was evaluated in detail
YaDiv requires Java and Java3D in order to work, and although these software were installed, the software did not work. The Lung segmentation tool by NIH-CIDI is only suitable for 2D segmentation. Therefore, only five software were evaluated for the segmentation of the lungs.
The Pulmonary toolkit/Github (Ptk): Ptk is a software suite for the analysis of 3D medical lung images, and is intended for academic and research use only. It requires MATLAB (version R2010 or later) as well as the MATLAB Image Processing toolbox. Some features of this software may additionally require a C++ compiler [2].
TurtleSeg: TurtleSeg provides accurate 3D image segmentation based on the 2D Turtlemap algorithm. The software is a non-open source; however, it was provided as a 32-bit trial version for 1 month after requesting a trial version [3].
Medical Imaging Interaction Toolkit (MITK): MITK is a free open-source software system which combines both the Insight Toolkit (ITK) and the Visualization Toolkit (VTK) with an application framework enabling several toolkits [4].
ITK-SNAP: ITK-SNAP is a software application that provides semi-automatic segmentation in 3D medical images using active contour methods, as well as manual delineation and image navigation. It is a free, open-source, and multi-platform software [5].
3D Slicer: 3D Slicer is a software platform for the analysis and visualization of medical images, and for research in image guided therapy. It enables registration and interactive segmentation, and is a free open source software [6].
2.2 Criteria used for evaluation
In evaluating the software, different parameters were analysed and compared for each software in order to carry out the evaluation in an objective manner. Functionality, usability, image quality and 3D export are regarded to be the most important parameters to survey a software for segmentation. This is because the software are expected to be able to segment the lungs exactly with minimal user-intervention, and has to enable the user to export segmented images of high quality in the form of images as well as 3D models for further processing and measurements.
A scoring system is introduced to make the evaluation process more quantitative and more accurate. For each of the above parameters, an evaluation standard is described which is further presented by points. In the end, the total score was given to each software.
2.2.1 Functionality
Functionality is defined in our study as the ability of the software to segment the lungs with minimal errors. The scale used for evaluating this parameter is:
4: Fully functional, the software can segment each lung on its own, the airways, and the lobes.
3: Functional, the software segments the lungs and the airways together, with the possibility of further segmenting/labelling by the user.
2: Functional, the software fully segments the airways and the lungs without labelling.
1: Functional, but the region segmented needs a lot of editing and smoothing by the user.
2.2.2 Usability
Usability indicates the level of easiness of using the software, and to what extent the software is able to run without intensive user intervention. The following scale has been used for the evaluation of the software:
4: Very easy to use, without user intervention
3: Easy to use, with very minimal user intervention
2: Easy to use, minimal user intervention required with some steps that need to be followed by the user
1: High user intervention is required
2.2.3 Quality of segmentation and 3D export
The quality of segmentation is regarded to be of high quality if the segmentation covers the needed lung region without leakage, and requires no further smoothing or editing. Heuristically, the following scale has been set:
4: high quality, 3: good quality, 2: satisfactory quality,
1: lower quality, requires a lot of editing and smoothing
The software which has more exporting file types of the generated 3D volume provides more options for further processing. One important requirement was the ability of the software to export Stereolithography (STL) files. The following scale was used for evaluation:
3: Can export STL and at least two other data types
2: Can export STL and one of remained types
1: Can export STL or one of remained types
2.3 Dataset
The lung segmentation and the evaluation of the software was performed by using a CT dataset of a male patient. The original dataset was 616 MB with 1219 slices, but it was reduced to 57.8 MB with only 59 slices in order to prevent the slowdown or crashing of software evaluated. The CT images were saved in DICOM form with 512 × 512 pixels.
3 Results
3.1 Functionality
Although all five evaluated software demonstrated ability to segment the lungs, there were variations in terms of to what extent is the software able to get a precise segmentation. Table 1 shows the results for the evaluation of this parameter.
Functionality of the evaluated software.
| Lungs seg. | Airway seg. | Lobes seg. | labelling | Score | |
|---|---|---|---|---|---|
| Ptk | + | + | + | + | 4 |
| TurtleSeg | + | − | − | − | 1 |
| MITK | + | + | − | − | 2 |
| ITK-SNAP | + | + | − | + | 3 |
| 3D Slicer | + | + | − | − | 2 |
3.2 Usability
Table 2 shows the results of evaluating usability of the different software, and reflects the facility to use each software for lung segmentation.
Usability of the evaluated software.
| Description | Score | |
|---|---|---|
| Ptk | Segmentation and other features are carried out automatically | 4 |
| TurtlSeg | User has to draw contours for each slice suggested by the software until the segmentation is satisfactory | 1 |
| MITK | The user has to only specify the threshold | 3 |
| ITK-SNAP | The user has to only add bubbles to the region that needs to be segmented, specify threshold and run the software | 2 |
| 3D Slicer | Volume has to be specified, and threshold has to be adjusted | 2 |
3.3 Segmentation quality and 3D export
3.3.1 Quality of segmented images
The quality of the segmentation was based on the quality of the generated 3D models. The segmentation results using the specified dataset are shown in Figure 1. Heuristically, the segmentation quality is stated in Table 3.
Segmented 3D lung volume are shown in anterior view: Ptk (top left), MITK (top right), 3D Slicer (middle left), ITK-SNAP (middle right), and TurtleSeg (bottom left).
Evaluating the quality of segmented images in each software.
| Ptk | TurtleSeg | MITK | ITK-SNAP | 3D slicer | |
|---|---|---|---|---|---|
| Description | High | Satisfactory | Good | Good | Lowera |
| Score | 4 | 2 | 3 | 3 | 1 |
aNeeds a lot of editing and smoothing.
3.3.2 3D export
Due to the need of exporting 3D models of the lungs for further measurements and applications, it is of importance to evaluate this parameter quantitatively, and for that Table 4 states the evaluation for this parameter.
Possible types of 3D export in each software.
| STL | VTK | Mesh file | XML | Score | |
|---|---|---|---|---|---|
| Ptk | − | − | + | − | 1 |
| TurtleSeg | + | + | − | − | 2 |
| MITK | + | + | − | − | 2 |
| ITK-SNAP | + | + | + | − | 3 |
| 3D Slicer | + | + | − | + | 3 |
3.4 Final results
For the final evaluation of the software, the individual evaluation of the important parameters shown in the previous sections were added all together for the total evaluation, as shown in Figure 2.
Total evaluation results for the software.
4 Discussion and conclusion
In this study, different parameters are defined to be subcategories of the comparison between five software. It enables the overall evaluation of the software as well as the individual evaluation of specific parameters like functionality or 3D export. As shown in Figure 2, Ptk obtains the highest evaluation score (13 points) in overall. Especially, Ptk performs much better in functionality and usability, given that the user only needs to import the images into the GUI and to press on the segmentation tool. It produces lung segmentation with the best quality and provides further utilities such as airway segmentation. However, it can also be observed that the possible export type of the generated 3D volume is limited to mesh data only.
MITK is also relatively easy to use, as the user needs to only perform the segmentation, and to follow it by adjusting the threshold. As for ITK-SNAP, it has similar evaluation score of MITK for functionality and usability, however, it performed better in functionality given that the user is able to further label and edit the generated 3D volume after segmentation is finished. ITK-SNAP involves more user interaction than MITK, and for that MITK is regarded to be more user friendly. The segmented lung regions in 3D can be exported in several types with MITK as well as ITK-SNAP.
As for the main limitation in our study, it was the use of only one CT dataset for the evaluation of the five software. The segmentation quality could be further evaluated with more CT data of patients with variant lung diseases. Moreover, the evaluation points ensure that the evaluation process is done in a quantitative and an objective manner.
Author’s Statement
Research funding: This work was supported by the BMBF grant no. 03FH038I3 (MOSES) and by the European Commission under grant no. FP7-PIRSES 318943 (eTime). Conflict of interest: Authors state no conflict of interest. Material and methods: Informed consent: Informed consent has been obtained from all individuals included in this study. Ethical approval: The research related to human use complies with all the relevant national regulations, institutional policies and was performed in accordance with the tenets of the Helsinki Declaration, and has been approved by the authors’ institutional review board or equivalent committee.
References
[1] Mansoor A, Bagci U, Foster B, Xu Z, Papadakis GZ, Folio LR, et al. Segmentation and image analysis of abnormal lungs at CT: current approaches, challenges, and future trends. INFORMATICS, July–August 2015.10.1148/rg.2015140232Search in Google Scholar PubMed PubMed Central
[2] Lung medical image analysis and visualisation software for Matlab. 2016. [Online]. Available from: https://github.com/tomdoel/pulmonarytoolkit. [Zugriff am 04 04 2016].Search in Google Scholar
[3] TurtleSeg. Oxipita Inc, 2010–2016. [Online]. Available from: http://www.turtleseg.org/index.php?page=home. [Zugriff am 04 04 2016].Search in Google Scholar
[4] German Cancer Research Center, 2016. [Online]. Available from: http://mitk.org/wiki/The_Medical_Imaging_Interaction_Toolkit_(MITK). [Zugriff am 04 04 2016].Search in Google Scholar
[5] Itk-SNAP, 2014. [Online]. Available from: http://www.itksnap.org/pmwiki/pmwiki.php. [Zugriff am 04 04 2016].Search in Google Scholar
[6] 3D Slicer Wiki pages, 2015. [Online]. Available from: http://wiki.slicer.org/slicerWiki/index.php/Main_Page. [Zugriff am 04 04 2016].Search in Google Scholar
©2016 Aseel Alnaser et al., licensee De Gruyter.
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
