Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Current Directions in Biomedical Engineering

Joint Journal of the German Society for Biomedical Engineering in VDE and the Austrian and Swiss Societies for Biomedical Engineering

Editor-in-Chief: Dössel, Olaf

Editorial Board: Augat, Peter / Buzug, Thorsten M. / Haueisen, Jens / Jockenhoevel, Stefan / Knaup-Gregori, Petra / Kraft, Marc / Lenarz, Thomas / Leonhardt, Steffen / Malberg, Hagen / Penzel, Thomas / Plank, Gernot / Radermacher, Klaus M. / Schkommodau, Erik / Stieglitz, Thomas / Urban, Gerald A.

CiteScore 2018: 0.47

Source Normalized Impact per Paper (SNIP) 2018: 0.377

Open Access
See all formats and pricing
More options …

Cerebral cortex classification by conditional random fields applied to intraoperative thermal imaging

Nico Hoffmann
  • Corresponding author
  • TU Dresden, Faculty of Medicine, Clinical Sensoring and Monitoring, Fetscherstraße 74, D-01062 Dresden, Germany, Phone: +49 351 458 6443, Fax: +49 351 458 6325
  • Email
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
/ Edmund Koch / Uwe Petersohn / Matthias Kirsch / Gerald Steiner
Published Online: 2016-09-30 | DOI: https://doi.org/10.1515/cdbme-2016-0105


Intraoperative thermal neuroimaging is a novel intraoperative imaging technique for the characterization of perfusion disorders, neural activity and other pathological changes of the brain. It bases on the correlation of (sub-)cortical metabolism and perfusion with the emitted heat of the cortical surface. In order to minimize required computational resources and prevent unwanted artefacts in subsequent data analysis workflows foreground detection is a important preprocessing technique to differentiate pixels representing the cerebral cortex from background objects. We propose an efficient classification framework that integrates characteristic dynamic thermal behaviour into this classification task to include additional discriminative features. The first stage of our framework consists of learning this representation of characteristic thermal time-frequency behaviour. This representation models latent interconnections in the time-frequency domain that cover specific, yet a priori unknown, thermal properties of the cortex. In a second stage these features are then used to classify each pixel’s state with conditional random fields. We quantitatively evaluate several approaches to learning high-level features and their impact to the overall prediction accuracy. The introduction of high-level features leads to a significant accuracy improvement compared to a baseline classifier.

Keywords: conditional random fields; 2D B-Spline; discrete wavelet transform; intraoperative thermal imaging; machine learning; representation learning

1 Introduction

Intraoperative thermal neuroimaging denotes an non-invasive and contactless imaging technique that measures the emitted temperature radiation of tissue during neurosurgical interventions. In neurosurgery, temperature gradients derive from heat transfers being caused by (sub-)cortical perfusion and neuronal activity. Gorbach et al. [1] and Shevelev et al. [2] showed the application of intraoperative thermography for tumour diagnostics. Steiner et. al. [3] demonstrated the detection of a ice-cold saline solution applied through a central line as a tool for perfusion diagnostics.

High-level feature representations are extracted from data in time-frequency domain so that they unveil time-dependent thermal behaviour of the human cortex. We further evaluate two common approaches to this feature representation learning task and finally analyse their influence to the overall performance. Latter is also compared to a simple baseline method.

2 Methods

Dynamic thermal signals can be described by a combination of several non-stationary characteristic signal components. These components originate from physiological sources as well as from noise. To approach this challenge subsequent signal analysis and classification is done in time-frequency domain to decompose the signal into its characteristic time-dependent components. As the physiological influences of cortical and subcortical perfusion as well as tissue composition are unknown and difficult to estimate there is no parametric model about cortical heat emission. For what reason we propose a machine learning framework that extracts empirical knowledge from learning data using linear feature representation schemes for pixelwise latent state prediction. The research related to human use has been complied with all the relevant national regulations, institutional policies and in accordance the tenets of the Helsinki Declaration, all intraoperative procedures in this work were approved by the Human Ethics Committee of the Technische Universität Dresden (no. EK 323122008). Informed consent has been obtained from all individuals included in this study.

2.1 Preliminaries

The global thermographic signal Tn×m respectively Tim of any recorded pixel 1 ≤ in and m time points is superimposed by several components such as physiological signals and thermic background noise Tbgn×m. In order to prevent learning background signals the latter component has to be separated from T prior to further analysis.

The approximation of background noise is done by exploiting its rather smooth nature by a two dimensional B-Spline model1 to estimate and remove the background signal. Suppose we are given two one-dimensional B-Spline bases Bxym1×n and Btm2×m consisting of local B-Spline functions given m1 and m2 knots to interpolate the data in spatial Bxy and temporal Bt domain. Adjacent local B-spline functions are joined at m1 resp. m1 knots with m1n and m2m. Let further vec(T)nm be the column stacked version of our measured temperature data T and B2d = BxyBt the tensor product ⊗ of both 1D B-Spline bases. The smooth background signal Tbg = B2d α^ can now be estimated by regressing vec(T) = B2d α using least squares given the normal equations:


In order to catch dynamic thermographic effects that correlate with the actual imaged object subsequent analysis is done in time-frequency domain by applying the Discrete Wavelet transform to the data. Given a frequency band (scale) j, index k and wavelet ψ, the background corrected wavelet coefficients ci( j, k) of pixel i read


The transformation is done in linear time using the fast wavelet transform based on pyramid algorithm. In the following we will drop ( j, k) to simplify the notation if both are clear from context.

2.2 Representation learning

Now we can proceed to finding a high-level representation of the thermal characteristics of our physiological signals. The representation should be robust to noise while preserving as much information as possible. In the following we will discuss two common approaches to learn a representation f(ci) of the wavelet transformed thermal data ci.

This validation dataset was subject to the PCA-CRF classifier and the results are shown in (B). Yellow colour indicate pixels with label cerebral cortex. Expert classification of the cerebral cortex is indicated by the blue solid line in both images.
Figure 1

This validation dataset was subject to the PCA-CRF classifier and the results are shown in (B). Yellow colour indicate pixels with label cerebral cortex. Expert classification of the cerebral cortex is indicated by the blue solid line in both images.

2.2.1 Bag of frequency words

A common approach to unveil representative features is the bag of words model. In terms of wavelet coefficients cim for n pixels, we solve


and get representative frequency words μjm that describe our training dataset. The k-Means algorithm provides one solution to this task, so that our dictionary μ is defined by μ=[μ1..μp]T. This allows us to represent the L2 distance of any new vector ci to each of the p words μjm (1 ≤ jp) in terms of our dictionary μ=[μ1,,μp]T by


with length p column vector of ones 1p and ⊗ denoting the Kronecker product.

2.2.2 PCA

In the past, PCA was applied to learn a representation of neuroimaging data [6]. Suppose we measure n correlated m-dimensional wavelet transformed time-series stored as n rows in matrix Cn×m. PCA now gives a set of m orthogonal vectors vjm and weights wjn with 1 ≤ jm so that ci=jwjvj. It can be shown that these orthogonal vectors vj correspond to the eigenvectors Φ=[v1,v2,,vm] of the empirical covariance matrix ΣC=CTCm×m so that ΣCΦ=ΦΛ. Each eigenvector vj now describes a characteristic high-level thermal behaviour of the data. The feature vector ci of pixel i is represented by


with Φp=[v1vp]T given dictionary size p.

2.3 Learning unary potentials

Suppose each pixel i has latent state yi𝒴 given f(ci). In case of BoW, f(ci) denotes the distance of ci to the learnt p representative frequency words μk (or analogously for the PCA approach vk) with 1 ≤ kp. The set of states 𝒴={fg,bg} now denotes the two classes of the pixels. Those pixels belonging to the cerebral cortex are classified as foreground (fg) the others as background (bg). The probability pi(yif(ci)) of each pixel belonging to state yi given the feature representation can now be learnt by a Random Forests (RF) [7] consisting of t decision trees. The trained RF yields a measure of certainty (probability) that an observed encoded representation f(ci) belongs to state yi as of p(yi = fgf(ci)) or p(yi = bgf(ci)). An ensemble decision regarding the actual state yi of the RF is obtained by averaging the output pik(yif(ci)) of each single decision tree k:


The probability distribution is estimated by training each of the t decision trees on a bootstrap sample of our training data. This strategy effectively prevents overfitting as discussed by [7].

2.4 Structured classification of thermal neuroimaging data

In order to incorporate structural information into the classification process we propose a conditional random field (CRF) model. In this framework, the posterior distribution p(Y|𝒞) of our latent variables yi𝒴 and observations f(ci)𝒞 can be formulated in terms of unary factors Ψ and pairwise factors ϕ on a undirected graph G = (V, E) as of


with V being the set of pixels and E the set of edges between adjacent pixels i and i′. By fulfilling the local Markov property, this function can be factored so that p(Y|𝒞) only depends on its direct neighbours. The unary potential Ψi(yi, f(ci)) encodes the prior probability learnt by the RF classifier whilst the pairwise potential ϕ(yi, yi′ enforces spatial homogeneity of the inferred labelling. Computations were further simplified by integrating the regular structure of the imaged data through a Potts model function


with indicator function 1yt=yt yielding one if yt=yt and zero otherwise. C is a smoothness penalty so that low values of C lead to rough solutions whilst the estimated labelling is getting smoother by increasing C. Since we are dealing with binary labels and Potts pairwise terms equation 2 is submodular what allows the application of very efficient inference method based on graph cuts. Minimizing equation 2 now corresponds to finding a maximum a-posteriori estimate of the labelling y.

3 Results and discussion

The results were achieved by evaluating five intraoperative thermal measurements of five different cases of length 1024 frames (20 s). For this purpose thermal neuroimaging data was acquired just after exposure of the cerebral cortex during neurosurgical tumour resections. Three out of these datasets were used for training plus testing and two for validation. The training sample consists of 30% equally sampled data points of all three test datasets equally representing cortical and background pixels. All computations were done on a workstation with dual Intel Xeon E5-2630, 128 GB Ram and Nvidia Geforce GTX Titan Black graphics card.

We evaluated both PCA and the Bag of Words model for feature representation learning. Baseline performance was quantified by training a RF on the average temperature distribution μT. The accuracy was computed by


given multiple labeled datasets. TC denotes the number of true cortical-, TB the number of true background- and C as well as B represent the number of cortex and background pixels. These labellings were acquired by a medical expert. The results of both approaches are shown in Tables 1 and 2.

Table 1

The Random Forest classifier with PCA and BoW feature learning significantly outperform the baseline classifier on three training/testing as well as two validation datasets.

Table 2

Extending the classifier by structural constraints further improves the overall accuracy of the baseline- and both feature learning approaches. For reasons of comparison the approaches were applied to the same data as described in Table 1.

By introducing high-level features into the classification process a significant improvement in accuracy between the baseline classifier and the extended version can be observed. The introduction of learnt high-level thermal features provides additional discriminative information resulting from characteristic thermal signatures. Structural information shrink the difference in accuracy between our baseline classifier and the extended ones. This is caused by temperature inhomogeneities correlating with tissue composition and perfusion which are compensated by the Potts model. In the present cortex classification task this behaviour seems favourable. Yet, in case of smaller objects like tumours or vessels further attention has to be paid to this behaviour in order to achieve reasonable true-positive rates.

4 Conclusion

Intraoperative thermal neuroimaging is a novel technique to image time-dependent cortical temperature variations during neurosurgical interventions. The main cause of temperature changes is cortical perfusion which is influenced by cell metabolism and tissue composition. The thermal processes of the exposed brain are not well understood, yet they provide valuable information to characterize tissue. In this work thermal process signatures are employed to improve the differentiation of pixels of the cerebral cortex to background pixels. For this purpose we propose a novel machine learning framework for analysis of intraoperative thermal neuroimaging data. The learning goal is to recognize dynamic temperature behaviour of the imaged human cortex. These high-level features are then incorporated into a subsequent tissue classification stage based on conditional random fields improving overall classification accuracy. In the future, this framework might enable a more fine-grain characterization of tissue composition based on its dynamic thermal behaviour.


The authors would like to thank all organizations and individuals that supported this research project.

Author’s Statement

Research funding: This work was supported by the European Social Fund (grant no. 100087783) and the Saxonian Ministry of Science and Art. Conflict of interest: Authors state no conflict of interest. Material and Methods: Informed consent: Informed consent has been obtained from all individuals included in this study. Ethical approval: The research related to human use complies with all the relevant national regulations, institutional policies and was performed in accordance with the tenets of the Helsinki Declaration, all intraoperative procedures in this work were approved by the Human Ethics Committee of the Technische Universität Dresden (no. EK 323122008).


  • [1]

    Gorbach AM, Heiss JD, Kopylev L, Oldfield EH. Intraoperative infrared imaging of brain tumors. J Neurosurg. 2004;101;960–9. Google Scholar

  • [2]

    Shevelev IA, Tsicalov EN, Gorbach AM, Budko KP, Sharaev GA. Thermoimaging of the brain. J Neurosci Methods. 1993;46:49–57. Google Scholar

  • [3]

    Steiner G, Sobottka SB, Koch E, Schackert G, Kirsch M. Intraoperative imaging of cortical cerebral perfusion by time-resolved thermography and multivariate data analysis. J Biomed Opt. 2011;16:016001–016001–6. Google Scholar

  • [4]

    Eilers PH, Currie ID, Durban M. Fast and compact smoothing on large multidimensional grids. Comput Stat Data An. 2006;50:61–76. Google Scholar

  • [5]

    Durban M, Currie I, Eilers P. Multidimensional P-spline mixed models : a unified approach to smoothing on large grids. 2006;1:2–5. Google Scholar

  • [6]

    Lu H, Plataniotis KN, Venetsanopoulos AN. MPCA: multilinear principal component analysis of tensor objects. IEEE Trans Neural Netw. 2008;19:18–39. Google Scholar

  • [7]

    Breiman L. Random forests. Machine Learning. 2001;45:5–32. Google Scholar


  • 1

    Further details regarding B-Spline regression can be found in [4], [5]. 

About the article

Published Online: 2016-09-30

Published in Print: 2016-09-01

Citation Information: Current Directions in Biomedical Engineering, Volume 2, Issue 1, Pages 475–478, ISSN (Online) 2364-5504, DOI: https://doi.org/10.1515/cdbme-2016-0105.

Export Citation

©2016 Nico Hoffmann et al., licensee De Gruyter.. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. BY-NC-ND 4.0

Comments (0)

Please log in or register to comment.
Log in