Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter July 9, 2021

Collocation based training of neural ordinary differential equations

Elisabeth Roesch, Christopher Rackauckas and Michael P. H. Stumpf

Abstract

The predictive power of machine learning models often exceeds that of mechanistic modeling approaches. However, the interpretability of purely data-driven models, without any mechanistic basis is often complicated, and predictive power by itself can be a poor metric by which we might want to judge different methods. In this work, we focus on the relatively new modeling techniques of neural ordinary differential equations. We discuss how they relate to machine learning and mechanistic models, with the potential to narrow the gulf between these two frameworks: they constitute a class of hybrid model that integrates ideas from data-driven and dynamical systems approaches. Training neural ODEs as representations of dynamical systems data has its own specific demands, and we here propose a collocation scheme as a fast and efficient training strategy. This alleviates the need for costly ODE solvers. We illustrate the advantages that collocation approaches offer, as well as their robustness to qualitative features of a dynamical system, and the quantity and quality of observational data. We focus on systems that exemplify some of the hallmarks of complex dynamical systems encountered in systems biology, and we map out how these methods can be used in the analysis of mathematical models of cellular and physiological processes.


Corresponding author: Michael P. H. Stumpf, School of BioSciences, Biosciences 4, The University of Melbourne, Royal Parade, Parkville, VIC3052, Australia; and School of Mathematics and Statistics, University of Melbourne, 813 Swanston Street, Parkville, VIC3010, Australia, E-mail:

Acknowledgment

We gratefully acknowledge discussions with members of the Theoretical Systems Biology group at University of Melbourne, Australia, and at Imperial College London, United Kingdom, as well as with the Julia community. The information, data, or work presented herein was funded in part by ARPA-E under award numbers DE-AR0001222 and DE-AR0001211, and NSF award number IIP-1938400. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.

  1. Author contribution: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

  2. Research funding: None declared.

  3. Conflict of interest statement: The authors declare no conflicts of interest regarding this article.

References

Álvarez et al., 2010 Álvarez, M., Luengo, D., Titsias, M., and Lawrence, N. (2010). Efficient multioutput Gaussian processes through variational inducing kernels. J. Mach. Learn. Res. 9: 25–32.Search in Google Scholar

Babtie, A.C., Kirk, P., and Stumpf, M.P.H. (2014). Topological sensitivity analysis for systems biology. Proc. Natl. Acad. Sci. U.S.A. 111: 18507–18512. https://doi.org/10.1073/pnas.1414026112.Search in Google Scholar

Baker, R.E., Peña, J.-M., Jayamohan, J., and Jérusalem, A. (2018). Mechanistic models versus machine learning, a fight worth fighting for the biological community?. Biol. Lett. 14: 20170660. https://doi.org/10.1098/rsbl.2017.0660.Search in Google Scholar

Che, Z., Purushotham, S., Cho, K., Sontag, D., and Liu, Y. (2018). Recurrent neural networks for multivariate time series with missing values. Sci. Rep. 8. https://doi.org/10.1038/s41598-018-24271-9.Search in Google Scholar

Chen, R.T.Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. (2019). Neural ordinary differential equations. arXiv.Search in Google Scholar

Crook, O.M., Gatto, L., and Kirk, P.D.W. (2019). Fast approximate inference for variable selection in dirichlet process mixtures, with an application to pan-cancer proteomics. Stat. Appl. Genet. Mol. Biol. 18: 20180065. https://doi.org/10.1515/sagmb-2018-0065.Search in Google Scholar

Dupont, E., Doucet, A., and Teh, Y.W. (2019). Augmented neural ODEs. arXiv.Search in Google Scholar

Durbin, J. and Koopman, S.J. (2012). Time series analysis by state space methods. Oxford statistical science series, 2nd ed. Oxford University Press, Oxford.10.1093/acprof:oso/9780199641178.001.0001Search in Google Scholar

Estakhroueieh, M., Nikravesh, S., and Gharibzadeh, S. (2014). ECG generation based on action potential using modified van der pol equation. Annu. Res. Rev. Biol. 4: 4259–4272. https://doi.org/10.9734/arrb/2014/11916.Search in Google Scholar

Gardiner, C. (2009). Stochastic methods: a handbook for the natural and social sciences. Springer, Berlin and Heidelberg.Search in Google Scholar

Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. J. Mach. Learn. Res. 9: 249–256.Search in Google Scholar

Gupta, A. and Khammash, M. (2014). Sensitivity analysis for stochastic chemical reaction networks with multiple time-scales. Electron. J. Probab. 19. https://doi.org/10.1214/ejp.v19-3246.Search in Google Scholar

Heinonen, M., Yildiz, C., Mannerström, H., Intosalmi, J., and Lähdesmäki, H. (2018). Learning unknown ODE models with Gaussian processes. arXiv.Search in Google Scholar

Innes, M. (2018). Flux: elegant machine learning with julia. J. Open Source Software 3: 602. https://doi.org/10.21105/joss.00602.Search in Google Scholar

Innes, M., Saba, E., Fischer, K., Gandhi, D., Rudilosso, M.C., Joy, N.M., Karmali, T., Pal, A., and Shah, V. (2018). Fashionable modelling with flux. arXiv.Search in Google Scholar

Jia, J. and Benson, A.R. (2019). Neural jump stochastic differential equations. arXiv.Search in Google Scholar

Jost, J. (2005). Dynamical systems examples of complex behaviour. Springer, Berlin and Heidelberg.Search in Google Scholar

Kersting, H., Krämer, N., Schiegg, M., Daniel, C., Tiemann, M., and Hennig, P. (2020). Differentiable likelihoods for fast inversion of ’likelihood-free’ dynamical systems. arXiv.Search in Google Scholar

Kirk, P., Thorne, T., and Stumpf, M.P. (2013). Model selection in systems and synthetic biology. Curr. Opin. Biotechnol. 24: 767–774. https://doi.org/10.1016/j.copbio.2013.03.012.Search in Google Scholar

Kirk, P.D.W., Babtie, A.C., and Stumpf, M.P.H. (2015). Systems biology (un)certainties. Science 350: 386–388. https://doi.org/10.1126/science.aac9505.Search in Google Scholar

Lakatos, E. and Stumpf, M.P.H. (2017). Control mechanisms for stochastic biochemical systems via computation of reachable sets. R. Soc. Open Sci. 4: 160790. https://doi.org/10.1098/rsos.160790.Search in Google Scholar

LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature 521: 436–444. https://doi.org/10.1038/nature14539.Search in Google Scholar

Leon, M., Woods, M.L., Fedorec, A.J.H., and Barnes, C.P. (2016). A computational method for the investigation of multistable systems and its application to genetic switches. BMC Syst. Biol. 10. https://doi.org/10.1186/s12918-016-0375-z.Search in Google Scholar

Liang, H. and Wu, H. (2008). Parameter estimation for differential equation models using a framework of measurement error in regression models. J. Am. Stat. Assoc. 103: 1570–1583. https://doi.org/10.1198/016214508000000797.Search in Google Scholar

Liepe, J., Filippi, S., Komorowski, M., and Stumpf, M.P.H. (2013). Maximizing the information content of experiments in systems biology. PLoS Comput. Biol. 9: e1002888. https://doi.org/10.1371/journal.pcbi.1002888.Search in Google Scholar

Liepe, J., Kirk, P., Filippi, S., Toni, T., Barnes, C.P., and Stumpf, M.P.H. (2014). A framework for parameter estimation and model selection from experimental data in systems biology using approximate bayesian computation. Nat. Protoc. 9: 439–456. https://doi.org/10.1038/nprot.2014.025.Search in Google Scholar

Liu, X., Xiao, T., Si, S., Cao, Q., Kumar, S., and Hsieh, C.-J. (2019). Neural SDE: stabilizing neural ODE networks with stochastic noise. arXiv.Search in Google Scholar

Milias-Argeitis, A., Summers, S., Stewart-Ornstein, J., Zuleta, I., Pincus, D., El-Samad, H., Khammash, M., and Lygeros, J. (2011). In silico feedback for in vivo regulation of a gene expression circuit. Nat. Biotechnol. 29: 1114–1116. https://doi.org/10.1038/nbt.2018.Search in Google Scholar

Murphy, K.P. (2012). Machine learning: a probabilistic perspective. MIT Press, Cambridge, Massachusetts and London, England.Search in Google Scholar

Rackauckas, C. and Nie, Q. (2017). DifferentialEquations.jl – a performant and feature-rich ecosystem for solving differential equations in julia. J. Open Res. Software 5. https://doi.org/10.5334/jors.151.Search in Google Scholar

Rackauckas, C., Innes, M., Ma, Y., Bettencourt, J., White, L., and Dixit, V. (2019). DiffEqFlux.jl–a julia library for neural differential equations. arXiv.Search in Google Scholar

Rackauckas, C., Ma, Y., Martensen, J., Warner, C., Zubov, K., Supekar, R., Skinner, D., and Ramadhan, A. (2020). Universal differential equations for scientific machine learning. arXiv.10.21203/rs.3.rs-55125/v1Search in Google Scholar

Rasmussen, C.E. and Williams, C.K.I. (2006). Gaussian processes for machine learning. MIT Press, Cambridge, Massachusetts and London, England.Search in Google Scholar

Roesch, E. and Stumpf, M.P.H. (2019). Parameter inference in dynamical systems with co-dimension 1 bifurcations. R. Soc. Open Sci. 6: 190747. https://doi.org/10.1098/rsos.190747.Search in Google Scholar

Rubanova, Y., Chen, R.T.Q., and Duvenaud, D. (2019). Latent ODEs for irregularly-sampled time series. arXiv.Search in Google Scholar

Schnoerr, D., Sanguinetti, G., and Grima, R. (2017). Approximation and inference methods for stochastic biochemical kinetics—a tutorial review. J. Phys. A: Math. Theor. 50: 093001. https://doi.org/10.1088/1751-8121/aa54d9.Search in Google Scholar

Scholes, N.S., Schnoerr, D., Isalan, M., and Stumpf, M.P.H. (2019). A comprehensive network atlas reveals that turing patterns are common but not robust. Cell Syst. 9: 243–257.e4. https://doi.org/10.1016/j.cels.2019.07.007.Search in Google Scholar

Silk, D., Barnes, C.P., Kirk, P.D.W., Kirk, P., Toni, T., Rose, A., Moon, S., Dallman, M.J., Stumpf, M.P.H., and Stumpf, M.P.H. (2011). Designing attractive models via automated identification of chaotic and oscillatory dynamical regimes. Nat. Commun. 2: 489. https://doi.org/10.1038/ncomms1496.Search in Google Scholar

Tankhilevich, E., Ish-Horowicz, J., Hameed, T., Roesch, E., Kleijn, I., Stumpf, M.P.H., and He, F. (2020). GpABC: a julia package for approximate bayesian computation with Gaussian process emulation. Bioinformatics 36: 3286–3287. https://doi.org/10.1093/bioinformatics/btaa078.Search in Google Scholar

Toni, T., Welch, D., Strelkowa, N., Ipsen, A., and Stumpf, M.P. (2008). Approximate bayesian computation scheme for parameter inference and model selection in dynamical systems. J. R. Soc. Interface 6: 187–202. https://doi.org/10.1098/rsif.2008.0172.Search in Google Scholar

Tyson, J.J., Chen, K.C., and Novák, B. (2003). Sniffers, buzzers, toggles and blinkers: dynamics of regulatory and signaling pathways in the cell. Curr. Opin. Cell Biol. 15: 221–231. https://doi.org/10.1016/s0955-0674(03)00017-6.Search in Google Scholar

Tzen, B. and Raginsky, M. (2019). Neural stochastic differential equations: deep latent Gaussian models in the diffusion limit. arXiv.Search in Google Scholar

Žurauskienė et al., 2014 Žurauskienė, J., Kirk, P., Thorne, T., and Stumpf, M.P. (2014). Bayesian non-parametric approaches to reconstructing oscillatory systems and the nyquist limit. Phys. A 407: 33–42.10.1016/j.physa.2014.03.069Search in Google Scholar

Received: 2020-04-06
Revised: 2021-05-03
Accepted: 2021-05-04
Published Online: 2021-07-09

© 2021 Walter de Gruyter GmbH, Berlin/Boston