Data augmentation for computed tomography angiography via synthetic image generation and neural domain adaptation

Data set and preprocessing

We used data available through the Rotterdam Coronary Artery Algorithm Evaluation Framework (RCAAEF) [3]. It comprises 78 artery segments from 18 distinct patients, acquired using three different scanners and annotated by three different observers.

For preprocessing, we followed [4], who converted annotated contour lines into voxel masks and subsequently transformed them into a cross-sectional view by using the provided artery centerlines. This representation was found to be preferable for U-Net segmentation [5] of tubular structures. Additionally, the volumes were normalized such that the grayvalues ranged between 0 and 255.

Generating moderately realistic synthetic data

For simulating 3D cross-sectional CTA artery representations, we needed to achieve high variability but still maintaining a realistic structure, i.e., a circular organic shape in the middle of each slice and variable background due to surrounding tissue. Both were modeled by parameterizing the radius by a sum of sine waves of different period length and random amplitude (e.g., Figure 2). These parameters were slightly changed for sequential slices to maintain inter-slice correlation. The resulting slices were stacked to form 3D data.

To further increase complexity and improve realism we simulated stenoses and plaques with varying degree of calcification. For these pathological changes, three shapes (D-shaped, centric and eccentric) were introduced and added randomly to the synthetic data. Finally, Gaussian noise was added and the resulting volume was filtered with a Gaussian filter.

Synthetic images and stylization

NST is an optimization method that uses a pre-trained CNN to extract statistics describing style and content of images [6]. Those are used to optimize an output image to match the content statistics of a content and the style statistics of a style reference image. For 3D data this optimization problem had to be solved for each slice individually, so we used only a single iteration to reduce the optimization time. For every synthetic image, a real artery segment was chosen randomly from the training set. This resulted in pairs of style and content images for every slice to which the style transfer was applied.

For the sake of completeness, we also implemented real-rime style transfer [7], that significantly speeds up the NST but only allows a single style image (FST).

FastPhotoStyle (FPS) [8] is another domain adaptation approach that is based on whitening and coloring transform (WCT) [9] and does not require further training. It could therefore be efficiently applied slice-wise with different style images using the previously described NST-algorithm.

For evaluation of image synthesis no widely accepted measure exists, so we decided on using multiple metrics to justify the synthetic components and to set the needed parameters for stylization. We tuned the parameters for the shape by measuring shape similarity of synthetic slices and histology images of coronary arteries based on Hu-Moments. Even though constructed for natural RGB images rather than medical images, Fréchet inception distance (FID) [10], which measures the distance between two data distributions via activations of pre-trained image classifiers, indicates at least a correlation regarding image realism. We extended the similarity analysis by multiscale structural similarity (MS-SSIM) [11], which compares contrast, luminance and structure (i.e., correlation) of two corresponding images on multiple scales. Lastly, the Histogram Intersection Similarity Method (HISM) was used to compare the color distributions of the 3D images without taking spatial attributes into account [12].

The same metrics were used to create the set up and evaluate the different style transfer algorithms.

Segmentation

For segmentation, a state-of-the-art 3D U-Net architecture proposed by [2] was used. A typical U-Net architecture consists of an equal amount of upsampling and downsampling layers [5]. Due to computational resource limitations, the number of feature maps was reduced to 4, 8, 16 and 32 in the four layers. We used the Generalized Dice Loss [13] function as it works well for imbalanced data. During training, random crops and flips were applied. Furthermore, only the labels of the second annotator were used to define a ground-truth and thus neglecting inter-observer variability. The training was performed on Google Colab with a batch size of six and was stopped after 3,000 epochs. We used the model with the best validation performance for further testing.

For evaluation of the segmentation results, we used the Generalized Dice Score [13] and Hausdorff distance. To analyze the effect of data augmentation on training sets of different sizes, we subsequently added artery segments of three patients, one from each scanner (see Figure 1). We performed three-fold cross-validation on the resulting subsets. Fully exploiting the limited total amount of data, we used dynamic sized test sets to compare different approaches trained on the same training subset. In order to compare the results of different training subsets, i.e., sets of different sizes, we introduced a fixed test set.

Figure 1:

Splitting of RCAAEF patient data (gray box) into training (green) and testing (blue) subsets of different sizes.

Figure 2:

Synthetic CTA slices. The first row shows four different processing steps of the developed algorithm. Second row shows an example original CTA slice from RCAAEF data set and three results of stylizing ‘random noise’-image

Two different analyses were carried out: the first one used training subset 1 (see Figure 1) containing data of three patients. It was then enlarged with different amounts of stylized and raw synthetic data. In a second study, 120 synthetic volumes were used to pre-train the segmentation network without stylization (noST) and stylized using FastPhotoStyle (3D-FPS) or neural style transfer (3D-NST). The resulting networks were evaluated with and without fine-tuning. To be able to fairly compare results of different models, we used a majority vote of the two remaining annotators to generate ground-truth for testing.

Synthetic images and stylization

For similarity analysis we successively applied deterioration techniques introduced in section “Generating moderately realistic synthetic data” to 30 synthetically created image volumes (Figure 2) and compared the results to 30 real volumes. Intervals for random value sampling were adjusted such that the similarity increased. Table 1 shows the corresponding results. The improving similarity metrics for each of the intermediate results of our algorithm justify the performed processing steps. Further we see that by applying style transfers to the ‘random noise’-images realism measured by all selected metrics improved significantly.

Segmentation results

Table 2 depicts the results of the pre-training study. The performance of 3D-NST on subset 1 (one network pre-trained) shows that networks only trained on the synthetic data already show promising results. Tuning the contracting part of the U-Net with real data (training subset 1 and 2) further improved the segmentation performance in terms of overlap and alignment. It is worth mentioning that on subset 2, this approach outperforms the annotators’ segmentation performance measured by GDS, whereas Hausdorff distances are inferior. When increasing the amount of original data further (training subset 3), we found that the positive effect of pre-training and fine-tuning diminishes.

In our second study, we found that also adding synthetic data to the training set improves the segmentation performance. The results can be found in Table 3. Over all three sets of synthetic data, the best performance was reached when having the same amount of original and synthetic data (12). We got the best performance for the network trained on not stylized synthetic data. With a GDS of 72.77±0.48% and a Hausdorff distance of 0.91±0.01mm, we even improved the results of pre-training.