Intraoperative thermal neuroimaging denotes an non-invasive and contactless imaging technique that measures the emitted temperature radiation of tissue during neurosurgical interventions. In neurosurgery, temperature gradients derive from heat transfers being caused by (sub-)cortical perfusion and neuronal activity. Gorbach et al.  and Shevelev et al.  showed the application of intraoperative thermography for tumour diagnostics. Steiner et. al.  demonstrated the detection of a ice-cold saline solution applied through a central line as a tool for perfusion diagnostics.
High-level feature representations are extracted from data in time-frequency domain so that they unveil time-dependent thermal behaviour of the human cortex. We further evaluate two common approaches to this feature representation learning task and finally analyse their influence to the overall performance. Latter is also compared to a simple baseline method.
Dynamic thermal signals can be described by a combination of several non-stationary characteristic signal components. These components originate from physiological sources as well as from noise. To approach this challenge subsequent signal analysis and classification is done in time-frequency domain to decompose the signal into its characteristic time-dependent components. As the physiological influences of cortical and subcortical perfusion as well as tissue composition are unknown and difficult to estimate there is no parametric model about cortical heat emission. For what reason we propose a machine learning framework that extracts empirical knowledge from learning data using linear feature representation schemes for pixelwise latent state prediction. The research related to human use has been complied with all the relevant national regulations, institutional policies and in accordance the tenets of the Helsinki Declaration, all intraoperative procedures in this work were approved by the Human Ethics Committee of the Technische Universität Dresden (no. EK 323122008). Informed consent has been obtained from all individuals included in this study.
The global thermographic signal respectively of any recorded pixel 1 ≤ i ≤ n and m time points is superimposed by several components such as physiological signals and thermic background noise . In order to prevent learning background signals the latter component has to be separated from T prior to further analysis.
The approximation of background noise is done by exploiting its rather smooth nature by a two dimensional B-Spline model1 to estimate and remove the background signal. Suppose we are given two one-dimensional B-Spline bases and consisting of local B-Spline functions given m1 and m2 knots to interpolate the data in spatial Bxy and temporal Bt domain. Adjacent local B-spline functions are joined at m1 resp. m1 knots with m1 ≪ n and m2 ≪ m. Let further be the column stacked version of our measured temperature data T and B2d = Bxy ⊗ Bt the tensor product ⊗ of both 1D B-Spline bases. The smooth background signal Tbg = B2d can now be estimated by regressing vec(T) = B2d α using least squares given the normal equations:
In order to catch dynamic thermographic effects that correlate with the actual imaged object subsequent analysis is done in time-frequency domain by applying the Discrete Wavelet transform to the data. Given a frequency band (scale) j, index k and wavelet ψ, the background corrected wavelet coefficients ci( j, k) of pixel i read
The transformation is done in linear time using the fast wavelet transform based on pyramid algorithm. In the following we will drop ( j, k) to simplify the notation if both are clear from context.
2.2 Representation learning
Now we can proceed to finding a high-level representation of the thermal characteristics of our physiological signals. The representation should be robust to noise while preserving as much information as possible. In the following we will discuss two common approaches to learn a representation f(ci) of the wavelet transformed thermal data ci.
2.2.1 Bag of frequency words
A common approach to unveil representative features is the bag of words model. In terms of wavelet coefficients for n pixels, we solve
and get representative frequency words that describe our training dataset. The k-Means algorithm provides one solution to this task, so that our dictionary μ is defined by . This allows us to represent the L2 distance of any new vector ci to each of the p words (1 ≤ j ≤ p) in terms of our dictionary by
with length p column vector of ones and ⊗ denoting the Kronecker product.
In the past, PCA was applied to learn a representation of neuroimaging data . Suppose we measure n correlated m-dimensional wavelet transformed time-series stored as n rows in matrix . PCA now gives a set of m orthogonal vectors and weights with 1 ≤ j ≤ m so that . It can be shown that these orthogonal vectors vj correspond to the eigenvectors of the empirical covariance matrix so that . Each eigenvector vj now describes a characteristic high-level thermal behaviour of the data. The feature vector ci of pixel i is represented by
with given dictionary size p.
2.3 Learning unary potentials
Suppose each pixel i has latent state given f(ci). In case of BoW, f(ci) denotes the distance of ci to the learnt p representative frequency words μk (or analogously for the PCA approach vk) with 1 ≤ k ≤ p. The set of states now denotes the two classes of the pixels. Those pixels belonging to the cerebral cortex are classified as foreground (fg) the others as background (bg). The probability pi(yi—f(ci)) of each pixel belonging to state yi given the feature representation can now be learnt by a Random Forests (RF)  consisting of t decision trees. The trained RF yields a measure of certainty (probability) that an observed encoded representation f(ci) belongs to state yi as of p(yi = fg—f(ci)) or p(yi = bg—f(ci)). An ensemble decision regarding the actual state yi of the RF is obtained by averaging the output pik(yi—f(ci)) of each single decision tree k:
The probability distribution is estimated by training each of the t decision trees on a bootstrap sample of our training data. This strategy effectively prevents overfitting as discussed by .
2.4 Structured classification of thermal neuroimaging data
In order to incorporate structural information into the classification process we propose a conditional random field (CRF) model. In this framework, the posterior distribution of our latent variables and observations can be formulated in terms of unary factors Ψ and pairwise factors ϕ on a undirected graph G = (V, E) as of
with V being the set of pixels and E the set of edges between adjacent pixels i and i′. By fulfilling the local Markov property, this function can be factored so that only depends on its direct neighbours. The unary potential Ψi(yi, f(ci)) encodes the prior probability learnt by the RF classifier whilst the pairwise potential ϕ(yi, yi′ enforces spatial homogeneity of the inferred labelling. Computations were further simplified by integrating the regular structure of the imaged data through a Potts model function
with indicator function yielding one if and zero otherwise. C is a smoothness penalty so that low values of C lead to rough solutions whilst the estimated labelling is getting smoother by increasing C. Since we are dealing with binary labels and Potts pairwise terms equation 2 is submodular what allows the application of very efficient inference method based on graph cuts. Minimizing equation 2 now corresponds to finding a maximum a-posteriori estimate of the labelling y.
3 Results and discussion
The results were achieved by evaluating five intraoperative thermal measurements of five different cases of length 1024 frames (20 s). For this purpose thermal neuroimaging data was acquired just after exposure of the cerebral cortex during neurosurgical tumour resections. Three out of these datasets were used for training plus testing and two for validation. The training sample consists of 30% equally sampled data points of all three test datasets equally representing cortical and background pixels. All computations were done on a workstation with dual Intel Xeon E5-2630, 128 GB Ram and Nvidia Geforce GTX Titan Black graphics card.
We evaluated both PCA and the Bag of Words model for feature representation learning. Baseline performance was quantified by training a RF on the average temperature distribution μT. The accuracy was computed by
given multiple labeled datasets. TC denotes the number of true cortical-, TB the number of true background- and C as well as B represent the number of cortex and background pixels. These labellings were acquired by a medical expert. The results of both approaches are shown in Tables 1 and 2.
By introducing high-level features into the classification process a significant improvement in accuracy between the baseline classifier and the extended version can be observed. The introduction of learnt high-level thermal features provides additional discriminative information resulting from characteristic thermal signatures. Structural information shrink the difference in accuracy between our baseline classifier and the extended ones. This is caused by temperature inhomogeneities correlating with tissue composition and perfusion which are compensated by the Potts model. In the present cortex classification task this behaviour seems favourable. Yet, in case of smaller objects like tumours or vessels further attention has to be paid to this behaviour in order to achieve reasonable true-positive rates.
Intraoperative thermal neuroimaging is a novel technique to image time-dependent cortical temperature variations during neurosurgical interventions. The main cause of temperature changes is cortical perfusion which is influenced by cell metabolism and tissue composition. The thermal processes of the exposed brain are not well understood, yet they provide valuable information to characterize tissue. In this work thermal process signatures are employed to improve the differentiation of pixels of the cerebral cortex to background pixels. For this purpose we propose a novel machine learning framework for analysis of intraoperative thermal neuroimaging data. The learning goal is to recognize dynamic temperature behaviour of the imaged human cortex. These high-level features are then incorporated into a subsequent tissue classification stage based on conditional random fields improving overall classification accuracy. In the future, this framework might enable a more fine-grain characterization of tissue composition based on its dynamic thermal behaviour.
The authors would like to thank all organizations and individuals that supported this research project.
Research funding: This work was supported by the European Social Fund (grant no. 100087783) and the Saxonian Ministry of Science and Art. Conflict of interest: Authors state no conflict of interest. Material and Methods: Informed consent: Informed consent has been obtained from all individuals included in this study. Ethical approval: The research related to human use complies with all the relevant national regulations, institutional policies and was performed in accordance with the tenets of the Helsinki Declaration, all intraoperative procedures in this work were approved by the Human Ethics Committee of the Technische Universität Dresden (no. EK 323122008).
Gorbach AM, Heiss JD, Kopylev L, Oldfield EH. Intraoperative infrared imaging of brain tumors. J Neurosurg. 2004;101;960–9. Google Scholar
Shevelev IA, Tsicalov EN, Gorbach AM, Budko KP, Sharaev GA. Thermoimaging of the brain. J Neurosci Methods. 1993;46:49–57. Google Scholar
Steiner G, Sobottka SB, Koch E, Schackert G, Kirsch M. Intraoperative imaging of cortical cerebral perfusion by time-resolved thermography and multivariate data analysis. J Biomed Opt. 2011;16:016001–016001–6. Google Scholar
Eilers PH, Currie ID, Durban M. Fast and compact smoothing on large multidimensional grids. Comput Stat Data An. 2006;50:61–76. Google Scholar
Durban M, Currie I, Eilers P. Multidimensional P-spline mixed models : a unified approach to smoothing on large grids. 2006;1:2–5. Google Scholar
Lu H, Plataniotis KN, Venetsanopoulos AN. MPCA: multilinear principal component analysis of tensor objects. IEEE Trans Neural Netw. 2008;19:18–39. Google Scholar
Breiman L. Random forests. Machine Learning. 2001;45:5–32. Google Scholar
About the article
Published Online: 2016-09-30
Published in Print: 2016-09-01
Citation Information: Current Directions in Biomedical Engineering, Volume 2, Issue 1, Pages 475–478, ISSN (Online) 2364-5504, DOI: https://doi.org/10.1515/cdbme-2016-0105.
©2016 Nico Hoffmann et al., licensee De Gruyter.. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. BY-NC-ND 4.0