Show Summary Details
More options …

# Computational Methods in Applied Mathematics

Editor-in-Chief: Carstensen, Carsten

Managing Editor: Matus, Piotr

4 Issues per year

IMPACT FACTOR 2017: 0.658

CiteScore 2017: 1.05

SCImago Journal Rank (SJR) 2017: 1.291
Source Normalized Impact per Paper (SNIP) 2017: 0.893

Mathematical Citation Quotient (MCQ) 2016: 0.75

Online
ISSN
1609-9389
See all formats and pricing
More options …

# Tensor Numerical Methods: Actual Theory and Recent Applications

Ivan Gavrilyuk
/ Boris N. Khoromskij
• Corresponding author
• Max-Planck-Institute for Mathematics in the Sciences, Inselstr. 22–26, 04103 Leipzig, Germany
• Email
• Other articles by this author:
Published Online: 2018-06-26 | DOI: https://doi.org/10.1515/cmam-2018-0014

## Abstract

Most important computational problems nowadays are those related to processing of the large data sets and to numerical solution of the high-dimensional integral-differential equations. These problems arise in numerical modeling in quantum chemistry, material science, and multiparticle dynamics, as well as in machine learning, computer simulation of stochastic processes and many other applications related to big data analysis. Modern tensor numerical methods enable solution of the multidimensional partial differential equations (PDE) in ${ℝ}^{d}$ by reducing them to one-dimensional calculations. Thus, they allow to avoid the so-called “curse of dimensionality”, i.e. exponential growth of the computational complexity in the dimension size d, in the course of numerical solution of high-dimensional problems. At present, both tensor numerical methods and multilinear algebra of big data continue to expand actively to further theoretical and applied research topics. This issue of CMAM is devoted to the recent developments in the theory of tensor numerical methods and their applications in scientific computing and data analysis. Current activities in this emerging field on the effective numerical modeling of temporal and stationary multidimensional PDEs and beyond are presented in the following ten articles, and some future trends are highlighted therein.

MSC 2010: 65F30; 65F50; 65N35; 65F10

Numerical treatment of the high-dimensional problems by using the traditional numerical methods suffers from the so-called “curse of dimensionality”. This exponential growth $O\left({n}^{d}\right)$ of the complexity of numerical algorithms in the dimension parameter d can be only slightly relaxed by parallel computations and high performance computing. The tensor numerical methods, bridging the multilinear algebra and nonlinear approximation theory, allow to reduce solution of multidimensional problems to one-dimensional calculations. They rely on an efficient separable representation of multivariate functions and operators on large ${n}^{\otimes d}$ grids, leading to algorithms of low computational cost that scale polynomially or linearly in the dimension parameter d. Thus, tensor methods may be understood as a discrete analogue of the separation of variables, which may be efficiently maintained at all steps of calculations. The target function can be the solution of some operator equation $Au=f$, in particular PDE, and it can be represented through the so-called solution operator $u=S\left(A\right)f$. In tensor numerical methods the entities u, f, A and also $S\left(A\right)$ are gainfully approximated in the low rank tensor formats.

Traditional methods of separable approximation combine the canonical, Tucker, as well as the matrix product states (MPS) formats, the latter known as the tensor train (TT) decomposition [22, 21]. The recent tensor methods in combination with exponentially accurate sinc-based approximations are proven to provide the $\mathcal{𝒪}\left(d\right)$ data-compression on a wide class of functions and operators [5, 6, 9, 7, 4]. The quantized-TT (QTT) tensor approximation of functions [13] makes it possible to solve high-dimensional PDEs in quantized tensor spaces, with the $\mathrm{log}$-volume complexity scaling in the full-grid size, i.e. $\mathcal{𝒪}\left(d\mathrm{log}n\right)$, instead of $\mathcal{𝒪}\left({n}^{d}\right)$.

At present, tensor numerical methods and multilinear algebra continue to expand rapidly to a wide range of theoretical and applied fields, see for example [11, 14, 10]. We also refer to the recent research monographs [15, 12], where the tensor numerical methods in scientific computing with the particular focus on multi-dimensional PDEs and electronic structure calculations have been presented. These trends are reflected also in the papers from the present issue of CMAM.

This special issue is a collection of papers which demonstrate that the tensor techniques allow to solve various hard theoretical and computational problems including approximation of multi-dimensional elliptic/parabolic PDEs. This issue includes ten invited contributions on theoretical analysis and applications of tensor-based numerical methods. These papers cover a broad range of topics including construction of computational schemes for steady-state and dynamical problems as well as for stochastic and parametric equations, separation rank estimates for classes of functions and operators, numerical simulations etc. Below we briefly describe the content of the special issue.

The goal of the paper [1] is the efficient numerical solution of stochastic eigenvalue problems. Such problems often lead to prohibitively high-dimensional systems with tensor product structure when discretized with the stochastic Galerkin method. The authors exploit this inherent tensor product structure to develop a globalized low-rank inexact Newton method with which they tackle the stochastic eigenvalue problem. The effectiveness of the solver is illustrated by numerical experiments.

The paper [2] deals with an algorithm for solution of high-dimensional evolutionary equations (ODEs and discretized time-dependent PDEs) in the TT decomposition, assuming that the solution and the right-hand side of the ODE admit such a decomposition with a low rank parameter. A linear ODE, discretized via one-step or Chebyshev differentiation schemes, turns into a large linear system. The tensor decomposition allows to solve this system for several time points simultaneously. In numerical experiments with the transport and the chemical master equations, the author demonstrates that the new method is faster than traditional time stepping and stochastic simulation algorithms.

The paper [3] examines a completely non-intrusive, sample-based method for the computation of functional low-rank solutions of high-dimensional parametric random PDEs which have become an area of intensive research in Uncertainty Quantification. In order to obtain a generalized polynomial chaos representation of the approximate stochastic solution, a novel black-box rank-adapted tensor reconstruction procedure is proposed. The performance of the described approach is illustrated with several numerical examples and compared to Monte Carlo sampling.

The authors of [8] consider the abstract differential equations of the heat and Schrödinger type and discuss various N-parametric approximations on the base of the Cayley transform and of the Laguerre expansion providing a sub-exponential accuracy, i.e. the accuracy of the order $\mathcal{𝒪}\left({e}^{-N}\mathrm{log}N\right)$. They propose a new approximation using the combination of the Gauss–Lobatto–Chebyshev interpolation and the Cayley transform and obtain a purely exponential accuracy of the order $\mathcal{𝒪}\left({e}^{-N}\right)$. The rank-structured tensor form of this approximation for a d-dimensional spatial operator coefficient results in an algorithm having a linear complexity in d.

The paper [16] study a dynamical low-rank approximation on the manifold of fixed-rank tensor trains, and analyze projection methods for the time integration of such problems. The authors prove error estimates for the explicit Euler method, amended with quasi-optimal projections to the manifold, under suitable approximability assumptions. Then they discuss the possibilities and difficulties with higher order explicit and implicit projected Runge–Kutta methods, in particular, the ways for limiting rank growth in the increments, and robustness with respect to small singular values.

The paper [17] deals with a new algorithm for spectral learning of Hidden Markov Models (HMM). In contrast to standard approach, the parameters of the HMM are not approximated directly, but through an estimate for the joint probability distribution. Using TT-format, the authors get an approximation by minimizing the Frobenius distance between the empirical joint probability distribution and tensors with low TT-ranks with core tensors normalization constraints. An algorithm for the solution of optimization problem that is based on the alternating least squares (ALS) approach is proposed and its fast version for sparse tensors is developed. The authors compare the performance of the proposed algorithm with the existing schemes and found that it is much more robust if the number of hidden states is overestimated.

The paper [18] describes advanced numerical tools for working with multivariate functions and for the analysis of large data sets. In particular, covariance matrices are crucial in spatio-temporal statistical tasks, but are often very expensive to compute and store, especially in 3D. Therefore, one can alternatively use a low-rank tensor formats, which reduce the computing and storage costs essentially. The authors apply the Tucker and canonical tensor decompositions to a family of Matérn-type radial functions with varying parameters and demonstrate theoretically and numerically that their tensor approximations exhibit exponentially fast convergence in the rank parameter, thus providing low computational complexity.

The paper [19] deals with a space-time isogeometric analysis scheme for the discretization of parabolic evolution equations with diffusion coefficients depending on both time and space variables. The problem is considered in a space-time cylinder in ${ℝ}^{d+1}$, with $d=2,3$ and is discretized using higher-order and highly-smooth spline spaces. This makes the matrix formation task very challenging from a computational point of view. The authors overcome this problem by introducing a low-rank decoupling of the operator into space and time components. Numerical experiments demonstrate the efficiency of this approach.

In [20] the authors propose an efficient algorithm to compute a low-rank approximation to the solution of so-called “Laplace-like” linear systems. The idea is to transform the problem into the frequency domain, and then to use cross approximation. In this case, we do not need to form explicit approximation to the inverse operator and can approximate the solution directly, which leads to reduced complexity. It is demonstrated that the proposed method is fast and robust by using it as a solver inside Uzawa iterative method for solving the Stokes problem.

The problem of approximately solving a system of univariate polynomials with one or more common roots and its coefficients corrupted by noise is studied in [23]. New Rayleigh quotient methods are proposed and evaluated for estimating the common roots. Using tensor algebra, reasonable starting values for the Rayleigh quotient methods can be computed. The new methods are compared to Gauss–Newton, solving an eigenvalue problem obtained from the generalized Sylvester matrix, and building a cluster among the roots of all polynomials. It is shown in a simulation study that Gauss–Newton and a new Rayleigh quotient method perform best, where the latter is more accurate when other roots than the true common roots are close together.

## Acknowledgements

We would like to thank Professor Carsten Carstensen, the editor-in-chief of the Journal of Computational Methods in Applied Mathematics, for his kind support of this special issue and for the useful comments on this overview paper. We thank the managing editor Professor Piotr Matus, and Dr. Almas Sherbaf for the effective assistance of the review and production process. We appreciate the authors of all articles published in this special issue for the excellent contributions as well as the reviewers for their work on refereeing the manuscripts.

## References

• [1]

P. Benner, A. Onwunta and M. Stoll, An inexact Newton–Krylov method for stochastic eigenvalue problems, Comput. Methods Appl. Math., to appear.  Google Scholar

• [2]

S. Dolgov, A tensor decomposition algorithm for large ODEs with conservation laws, Comput. Methods Appl. Math., to appear.  Google Scholar

• [3]

M. Eigel, J. Neumann, R. Schneider and S. Wolf, Non-intrusive tensor reconstruction for high dimensional random PDEs, Comput. Methods Appl. Math., to appear.  Google Scholar

• [4]

I. Gavrilyuk, Super exponentially convergent approximation to the solution of the Schrödinger equation in abstract setting, Comput. Methods Appl. Math. 10 (2010), no. 4, 345–358.  Google Scholar

• [5]

I. Gavrilyuk, W. Hackbusch and B. Khoromskij, Data-sparse approximation to the operator-valued functions of elliptic operators, Math. Comp. 73 (2004), 1297–1324.  Google Scholar

• [6]

I. Gavrilyuk, W. Hackbusch and B. Khoromskij, Tensor-product approximation to elliptic and parabolic solution operators in higher dimensions, Computing 74 (2005), 131–157.

• [7]

I. Gavrilyuk and B. Khoromskij, Quantized-TT-Cayley transform for computing the dynamics and the spectrum of high-dimensional Hamiltonians, Comput. Methods Appl. Math. 11 (2011), no. 3, 273–290.  Google Scholar

• [8]

I. Gavrilyuk and B. Khoromskij, Quasi-optimal rank-structured approximation to multidimensional parabolic problems by Cayley transform and Chebyshev interpolation, Comput. Methods Appl. Math., to appear.  Google Scholar

• [9]

I. Gavrilyuk, V. Makarov and V. Vasylyk, Exponentially Convergent Algorithms for Abstract Differential Equations, Birkhäuser, Basel, 2011.

• [10]

L. Grasedyck, D. Kressner and C. Tobler, A literature survey of low-rank tensor approximation techniques, GAMM-Mitt. 36 (2013), no. 1, 53–78.

• [11]

W. Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Springer, Berlin, 2012.  Google Scholar

• [12]

V. Khoromskaia and B. N. Khoromskij, Tensor Numerical Methods in Computational Quantum Chemistry, De Gruyter, Berlin, 2018.  Google Scholar

• [13]

B. N. Khoromskij, $O\left(d\mathrm{log}N\right)$-quantics approximation of N-d tensors in high-dimensional numerical modeling, Constr. Approx. 34 (2011), no. 2, 257–289.  Google Scholar

• [14]

B. N. Khoromskij, Tensors-structured numerical methods in scientific computing: Survey on recent advances, Chemometr. Intell. Lab. Syst. 110 (2012), 1–19.

• [15]

B. N. Khoromskij, Tensor Numerical Methods in Scientific Computing, De Gruyter, Berlin, 2018.  Google Scholar

• [16]

E. Kieri and B. Vandereycken, Projection methods for dynamical low-rank approximation of high-dimensional problems, Comput. Methods Appl. Math., to appear.  Google Scholar

• [17]

M. A. Kuznetsov and I. V. Oseledets, Tensor train spectral method for learning of hidden Markov models (HMM), Comput. Methods Appl. Math., to appear.  Google Scholar

• [18]

A. Litvinenko, D. Keyes, V. Khoromskaia, B. Khoromskij and H. Matthies, Tucker tensor analysis of Matérn functions in spatial statistics, Comput. Methods Appl. Math., to appear.  Google Scholar

• [19]

A. Mantzaflaris, F. Scholz and I. Toulopoulos, Low-rank space-time decoupled isogeometric analysis for parabolic problems with varying coefficients, Comput. Methods Appl. Math., to appear.  Google Scholar

• [20]

E. A. Muravleva and I. V. Oseledets, Approximate solution of linear systems with Laplace-like operators via cross approximation in the frequency domain, Comput. Methods Appl. Math., to appear.  Google Scholar

• [21]

I. V. Oseledets, Tensor train decomposition, SIAM J. Sci. Comput. 33 (2011), no. 5, 2295–2317.

• [22]

I. V. Oseledets and E. E. Tyrtyshnikov, Breaking the curse of dimensionality, or how to use SVD in many dimensions, SIAM J. Sci. Comput. 31 (2009), 3744–3759.

• [23]

A. Stegeman and L. De Lathauwer, Rayleigh quotient methods for estimating common roots of noisy univariate polynomials, Comput. Methods Appl. Math., to appear.  Google Scholar

Revised: 2018-06-01

Accepted: 2018-06-06

Published Online: 2018-06-26

Citation Information: Computational Methods in Applied Mathematics, ISSN (Online) 1609-9389, ISSN (Print) 1609-4840,

Export Citation

© 2018 Walter de Gruyter GmbH, Berlin/Boston.