Abstract
Nonlinear least squares iterative solver is considered for real-valued sufficiently smooth functions. The algorithm is based on successive solution of orthogonal projections of the linearized equation on a sequence of appropriately chosen low-dimensional subspaces. The bases of the latter are constructed using only the first-order derivatives of the function. The technique based on the concept of the limiting stepsize along normalized direction (developed earlier by the author) is used to guarantee the monotone decrease of the nonlinear residual norm. Under rather mild conditions, the convergence to zero is proved for the gradient and residual norms. The results of numerical testing are presented, including not only small-sized standard test problems, but also larger and harder examples, such as algebraic problems associated with canonical decomposition of dense and sparse 3D tensors as well as finite-difference discretizations of 2D nonlinear boundary problems for 2nd order partial differential equations.
References
[1] Armijo, L. Minimization of functions having Lipschitz continuous first partial derivatives. Pacific Journal of mathematics, 1966, 16(1), 1–310.2140/pjm.1966.16.1Search in Google Scholar
[2] Bellavia, S., Bertaccini, D., Morini, B. Nonsymmetric preconditioner updates in Newton-Krylov methods for nonlinear systems. SIAM Journal on Scientific Computing, 2011, 33(5), 2595–261910.1137/100789786Search in Google Scholar
[3] Bellavia, S., Gratton, S., Riccietti, E. A Levenberg-Marquardt method for large nonlinear least-squares problems with dynamic accuracy in functions and gradients. Numerische Mathematik, 2018, 140(3), 791–82510.1007/s00211-018-0977-zSearch in Google Scholar
[4] Bellavia, S., Morini, B. Strong local convergence properties of adaptive regularized methods for nonlinear least squares. IMA Journal of Numerical Analysis, 2015, 35(2), 947–96810.1093/imanum/dru021Search in Google Scholar
[5] Buckley, A., Lenir, A. QN-like Variable Storage Conjugate Gradients. Mathematical Programming, 1983, 27, 155–17510.1007/BF02591943Search in Google Scholar
[6] Brent, R. P. Algorithms for matrix multiplication (No. STAN-CS-70-157). Stanford Univ. CA Dept. of Computer Science, 197010.21236/AD0705509Search in Google Scholar
[7] Hestenes M. R., Stiefel E., Methods of conjugate gradients for solving linear systems, J. of Research of the National Bureau of Standards, 1952, 49 (6), 409–43610.6028/jres.049.044Search in Google Scholar
[8] Heule, M. J., Kauers, M., Seidl, M. New ways to multiply 3x3-matrices, 2019, arXiv preprint arXiv:1905.10192Search in Google Scholar
[9] Kaporin, I. E. Esimating global convergence of inexact Newton methods via limiting stepsize along normalized direction, Rep.9329, July 1993, Department of Mathematics, Catholic University of Nijmegen, Nijmegen, The Netherlands, 8p.Search in Google Scholar
[10] Kaporin, I. E. The use of preconditioned Krylov subspaces in conjugate gradient type methods for the solution of nonlinear least square problems, (Russian) Vestnik Mosk. Univ., Ser.15 (Computational Math. and Cybernetics) 1995, N 3, 26–31Search in Google Scholar
[11] Kaporin, I. E., Axelsson, O. On a class of nonlinear equation solvers based on the residual norm reduction over a sequence of aflne subspaces, SIAM J. Sci. Comput., 1994, 16(1), 228–24910.1137/0916015Search in Google Scholar
[12] Kaporin I. Preconditioned Subspace Descent Methods for the Solution of Nonlinear Systems of Equations, In: Proc. X International Conference on Optimization and Applications (OPTIMA 2019) (M. Jacimovic et al., eds.) Communications in Computer and Information Science (CCIS), Springer Nature Switzerland AG, 2019.10.1007/978-3-030-38603-0_13Search in Google Scholar
[13] Karimi, S., Vavasis, S. Conjugate gradient with subspace optimization, 2012, arXiv preprint arXiv:1202.1479v1Search in Google Scholar
[14] Kazeev, V. A., Tyrtyshnikov, E. E. Structure of the Hessian matrix and an economical implementation of Newton’s method in the problem of canonical approximation of tensors. Computational Mathematics and Mathematical Physics, 2010, 50(6), 927–94510.1134/S0965542510060011Search in Google Scholar
[15] Laderman, J. D. A noncommutative algorithm for multiplying 3x3 matrices using 23 multiplications. Bulletin of the American Mathematical Society, 1976, 82(1), 126–12810.1090/S0002-9904-1976-13988-2Search in Google Scholar
[16] Luksan, L., Matonoha, C., Vlcek, J. Problems for Nonlinear Least Squares and Nonlinear Equations, Technical Report V-1259, ICS AS CR, Prague, 2018Search in Google Scholar
[17] More, J. J., Garbow, B. S., Hillstrom, K. E. Testing unconstrained optimization software. Argonne National Laboratory, Applied Mathematics Division, Technical Memorandum No.324 (July 1978), 96 pp.10.2172/6650344Search in Google Scholar
[18] Oseledets, I. V., Savostyanov, D. V. Minimization methods for approximating tensors and their comparison. Computational Mathematics and Mathematical Physics, 2006, 46(10), 1641–165010.1134/S0965542506100022Search in Google Scholar
[19] Sterck, H. D., Winlaw, M. A nonlinearly preconditioned conjugate gradient algorithm for rank-R canonical tensor approximation. Numerical Linear Algebra with Applications, 2015, 22(3), 410–43210.1002/nla.1963Search in Google Scholar
[20] Sterck, H. D., Miller, K. An adaptive algebraic multigrid algorithm for low-rank canonical tensor decomposition. SIAM Journal on Scientific Computing, 2013, 35(1), B1–B2410.1137/110855934Search in Google Scholar
[21] Strassen, V. Gaussian elimination is not optimal, Numerische Mathematik, 1969, 13(4), 354–35610.1007/BF02165411Search in Google Scholar
[22] Tichavsky, P., Phan, A. H., Cichocki, A. Numerical CP decomposition of some diflcult tensors. Journal of Computational and Applied Mathematics, 2017, 317, 362–37010.1016/j.cam.2016.12.007Search in Google Scholar
[23] Toint, Ph. L. Some Numerical Results Using a Sparse Matrix Updating Formula in Unconstrained Optimization, Mathematics of Computation 1978, 32(143), 839–85110.1090/S0025-5718-1978-0483452-7Search in Google Scholar
[24] Yuan, Y. X. Recent advances in numerical methods for nonlinear equations and nonlinear least squares, Numerical algebra, control and optimization, 2011, 1(1), 15–3410.3934/naco.2011.1.15Search in Google Scholar
© 2020 Igor Kaporin, published by De Gruyter
This work is licensed under the Creative Commons Attribution 4.0 International License.