Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Statistical Applications in Genetics and Molecular Biology

Editor-in-Chief: Sanguinetti, Guido

6 Issues per year

IMPACT FACTOR 2017: 0.812
5-year IMPACT FACTOR: 1.104

CiteScore 2017: 0.86

SCImago Journal Rank (SJR) 2017: 0.456
Source Normalized Impact per Paper (SNIP) 2017: 0.527

Mathematical Citation Quotient (MCQ) 2017: 0.04

See all formats and pricing
More options …
Volume 13, Issue 5


Volume 10 (2011)

Volume 9 (2010)

Volume 6 (2007)

Volume 5 (2006)

Volume 4 (2005)

Volume 2 (2003)

Volume 1 (2002)

Quantifying the multi-scale performance of network inference algorithms

Chris J. Oates
  • Corresponding author
  • Department of Statistics, Zeeman Building, University of Warwick, Gibbet Hill Road, Coventry, CV4 7AL, UK
  • Email
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
/ Richard Amos / Simon E.F. Spencer
  • Department of Statistics, Zeeman Building, University of Warwick, Gibbet Hill Road, Coventry, CV4 7AL, UK
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
Published Online: 2014-08-05 | DOI: https://doi.org/10.1515/sagmb-2014-0012


Graphical models are widely used to study complex multivariate biological systems. Network inference algorithms aim to reverse-engineer such models from noisy experimental data. It is common to assess such algorithms using techniques from classifier analysis. These metrics, based on ability to correctly infer individual edges, possess a number of appealing features including invariance to rank-preserving transformation. However, regulation in biological systems occurs on multiple scales and existing metrics do not take into account the correctness of higher-order network structure. In this paper novel performance scores are presented that share the appealing properties of existing scores, whilst capturing ability to uncover regulation on multiple scales. Theoretical results confirm that performance of a network inference algorithm depends crucially on the scale at which inferences are to be made; in particular strong local performance does not guarantee accurate reconstruction of higher-order topology. Applying these scores to a large corpus of data from the DREAM5 challenge, we undertake a data-driven assessment of estimator performance. We find that the “wisdom of crowds” network, that demonstrated superior local performance in the DREAM5 challenge, is also among the best performing methodologies for inference of regulation on multiple length scales.

This article offers supplementary material which is provided at the end of the article.

Keywords: multi-scale scores; network inference; performance assessment


  • Banerjee, A. and J. Jost (2009): “Graph spectra as a systematic tool in computational biology,” Discrete Appl. Math., 157, 2425–2431.Web of ScienceGoogle Scholar

  • Berkhin, P. (2005): “A survey on pagerank computing,” Internet Mathematics, 2, 73–120.Google Scholar

  • Breitkreutz, D., L. Hlatky, E. Rietmanc and J. A. Tuszynski (2012): “Molecular signaling network complexity is correlated with cancer patient survivability,” Proc. Natl. Acad. Sci. USA 109, 9209–9212.Google Scholar

  • Cantone, I., L. Marucci, F. Iorio, M. A. Ricci, V. Belcastro, M. Bansal, S. Santini, M. di Bernardo, D. di Bernardo and M. P. Cosma (2009): “A yeast synthetic network for in vivo assessment of reverse-engineering and modeling approaches,” Cell 137, 172–181.Web of ScienceGoogle Scholar

  • Chuang, H. Y., E. Lee and T. Ideker (2007): “Network-based classification of breast cancer metastasis,” Mol. Syst. Biol. 3, 140.Web of ScienceGoogle Scholar

  • Dash, D. (2003): “Caveats for causal reasoning with equilibrium models,” PhD thesis, Intelligent Systems Program, University of Pittsburgh.Google Scholar

  • Davis, J. and M. Goadrich (2006): “The relationship between Precision-Recall and ROC curves.” Proceedings of the 23rd International Conference on Machine Learning. ACM Press, New York, pp. 233–240.Google Scholar

  • De Jongh, M. and M. J. Druzdzel (2009): “A comparison of structural distance measures for causal Bayesian network models.” In: Klopotek, M., Przepiorkowski, A., Wierzchon, S.T., Trojanowski, K. (Eds.), Recent Advances in Intelligent Information Systems, Academic Publishing House EXIT, Warsaw, pp. 443–456.Google Scholar

  • Drummond, C. and R. C. Holte (2004): “What ROC curves can’t do (and cost curves can).” Proc. 1st Workshop ROC Analysis in AI, pp. 19–26.Google Scholar

  • Fawcett, T. (2006): “An introduction to ROC analysis,” Pattern Recogn, 27, 861–874.CrossrefGoogle Scholar

  • Feiglin, A., A. Hacohen, A. Sarusi, J. Fisher, R. Unger and Y. Ofran (2012): “Static network structure can be used to model the phenotypic effects of perturbations in regulatory networks,” Bioinformatics, 28, 2811–2818.PubMedWeb of ScienceCrossrefGoogle Scholar

  • Feizi, S., D. Marbach, M. Médard and M. Kellis (2013): “Network deconvolution as a general method to distinguish direct dependencies in networks,” Nat. Biotechnol., 31, 726–733.Web of ScienceGoogle Scholar

  • Filosi, M., R. Visintainer, S. Riccadonna, G. Jurman and C. Furlanello (2014): “Stability indicators in network reconstruction,” PloS One, 9, e89815.Google Scholar

  • Goadrich, M., L. Oliphant and J. Shavlik (2004): “Learning ensembles of first-order clauses for recall-precision curves: A case study in biomedical information extraction.” Proceedings of the 14th International Conference on Inductive Logic Programming (ILP).Google Scholar

  • Heiser, L. M., A. Sadanandam, W. L. Kuo, S. C. Benz, T. C. Goldstein, S. Ng, W. J. Gibb, N. J. Wang, S. Ziyad, F. Tong, N. Bayani, Z, Hu, J. I. Billig, A. Dueregger, S. Lewis, L. Jakkula, J. E. Korkola, S. Durinck, F. Pepin, Y. Guan, E. Purdom, P. Neuvial, H. Bengtsson, K. W. Wood, P. G. Smith, L. T. Vassilev, B. T. Hennessy, J. Greshock, K. E. Bachman, M. A. Hardwicke, J. W. Park, L. J. Marton, D. M. Wolf, E. A. Collisson, R. M. Neve, G. B. Mills, T. P. Speed, H. S. Feiler, R. F. Wooster, D. Haussler, J. M. Stuart, J. W. Gray and P. T. Spellman (2012): “Subtype and pathway specific responses to anticancer compounds in breast cancer,” Proc. Natl. Acad. Sci. USA, 109, 2724–2729.CrossrefGoogle Scholar

  • Hill, S. M., Y. Lu, J. Molina, L. M. Heiser, P. T. Spellman, T. P. Speed, J. W. Gray, G. B. Mills and S. Mukherjee (2012): “Bayesian inference of signaling network topology in a cancer cell line,” Bioinformatics, 28, 2804–2810.CrossrefWeb of ScienceGoogle Scholar

  • Iwasaki, Y. and H. A. Simon (1994): “Causality and model abstraction,” Artif. Intell., 67, 143–194.Google Scholar

  • Jeong, H., B. Tombor, R. Albert, Z. N. Oltvai and A. -L. Barabási (2000): “The large-scale organization of metabolic networks,” Nature, 407, 651–654.Google Scholar

  • Johannes, M., J. C. Brase, H. Fröhlich, S. Gade, M. Gehrmann, M. Fälth, H. Sültmann and T. Beissbarth (2010): “Integration of pathway knowledge into a reweighted recursive feature elimination approach for risk stratification of cancer patients,” Bioinformatics, 26, 2136–2144.CrossrefWeb of SciencePubMedGoogle Scholar

  • Jurman, G., S. Riccadonna, R. Visintainer and C. Furlanello (2011a): “Biological network comparison via Ipsen-Mikhailov distance,” arXiv, 1109.0220.Google Scholar

  • Jurman, G., R. Visintainer and C. Furlanello (2011b): “An introduction to spectral distances in networks,” Fr. Art. In. 226, 227–234.Google Scholar

  • Lakhina, A., K. Papagiannaki, M. Crovella, C. Diot, E. D. Kolaczyk and N. Taft (2004): “Structural analysis of network traffic flows.” Proceedings of International Conference on Measurement and Modeling of Computer Systems, ACM, New York, USA, pp. 61–72.Google Scholar

  • Maathuis, M. H., D. Colombo, M. Kalisch and P. Bühlmann (2010): “Predicting causal effects in large-scale systems from observational data,” Nat. Methods, 7, 247–248.Web of ScienceGoogle Scholar

  • Marbach, D., T. Schaffter, C. Mattiussi and D. Floreano (2009): “Generating realistic in silico gene networks for performance assessment of reverse engineering methods,” J. Comp. Biol., 16, 229–239.Web of ScienceGoogle Scholar

  • Marbach, D., R. J. Prill, T. Schaffter, C. Mattiussi, D. Floreano and G. Stolovitzky (2010): “Revealing strengths and weaknesses of methods for gene network inference,” Proc. Natl. Acad. Sci. USA, 107, 6286–6291.CrossrefGoogle Scholar

  • Marbach, D., J. C. Costello, R. Küffner, N. M. Vega, R. J. Prill, D. M. Camacho, K. R. Allison, M. Kellis, J. J. Collins and G. Stolovitzky (2012): “Wisdom of crowds for robust gene network inference,” Nat. Methods, 9, 796–804.Web of ScienceGoogle Scholar

  • Milenković, T., F. Ioannis, L. Michael and N. Pržulj (2009): “Optimized null models for protein structure networks,” PLoS One, 4, e5967.Google Scholar

  • Morrison, J. L., R. Breitling, D. J. Higham and D. R. Gilbert (2005): “GeneRank: using search engine technology for the analysis of microarray experiments,” BMC Bioinformatics, 6, 233.CrossrefPubMedGoogle Scholar

  • Nelander, S., W. Wang, B. Nilsson, Q. B. She, C. Pratilas, N. Rosen, P. Gennemark and C. Sander (2008): “Models from experiments: combinatorial drug perturbations of cancer cells,” Mol. Syst. Biol., 4, 216.PubMedWeb of ScienceGoogle Scholar

  • Oates, C. and S. Mukherjee (2012): “Network inference and biological dynamics,” Ann. Appl. Stat., 6, 1209–1235.Web of ScienceCrossrefGoogle Scholar

  • Oates, C.J., J. Korkola, J. W. Gray and S. Mukherjee (2014a): “Joint estimation of multiple related biological networks,” Ann. Appl. Stat., to appear.Google Scholar

  • Oates, C. J., F. Dondelinger, N. Bayani, J. Korola, J. W. Gray and S. Mukherjee (2014b): “Causal network inference using biochemical kinetics,” Bioinformatics, to appear.Web of ScienceGoogle Scholar

  • Page, L., S. Brin, R. Motwani and T. Winograd (1999): “The pagerank citation ranking: bringing order to the web,” Technical Report, Stanford InfoLab.Google Scholar

  • Pearl, J. (2000): Causality: models, reasoning, and inference. Cambridge University Press, Cambridge, UK.Google Scholar

  • Peters, J. and P. Bühlmann (2013): “Structural Intervention Distance (SID) for evaluating causal graphs,” arXiv, 1306.1043.Google Scholar

  • Prill, R. J., D. Marbach, J. Saez-Rodriguez, P. K. Sorger, L. G. Alexopoulos, X. Xue, N. D. Clarke, G. Altan-Bonnet and G. Stolovitzky (2010): “Towards a rigorous assessment of systems biology models: The DREAM3 challenges,” PLoS One, 5, e9202.Google Scholar

  • Prill, R. J., J. Saez-Rodriguez, L. G. Alexopoulos, P. K. Sorger and G. Stolovitzky (2011): “Crowdsourcing network inference: the DREAM predictive signaling network challenge,” Sci. Signal, 4(189), mr7.Google Scholar

  • Scutari, M. and R. Nagarajan (2013): “Identifying significant edges in graphical models of molecular networks,” Artif. Intell. Med., 57, 207–217.Web of ScienceCrossrefGoogle Scholar

  • Shrivastava, A. and P. Li (2014): “A new space for comparing graphs,” arXiv, 1404.4644.Google Scholar

  • Simon, N. and R. Tibshirani (2012): Comment on “Detecting novel associations in large datasets” by Reshef et al. Technical Report, Stanford, CA, USA.Google Scholar

  • Städler, N. and S. Mukherjee (2014): “Multivariate gene-set testing based on graphical models,” Biostatistics, to appear.Google Scholar

  • Tong, A. H. Y., G. Lesage, G. D. Bade, H. Ding, H. Xu, X. Xin, J. Young, G. F. Berriz, R. L. Brost, M. Chang, Y. Q. Chen, X. Cheng, G. Chua, H. Friesen, D. S. Goldberg, J. Haynes, C. Humphries, G. He, S. Hussein, L. Ke, N. Krogan, Z. Li, J. N. Levinson, H. Lu, P. Ménard, C. Munyana, A. B. Parsons, O. Ryan, R. Tonikian, T. Roberts, A. M. Sdicu, J. Shapiro, B. Sheikh, B. Suter, S. L. Wong, L. V. Zhang, H. Zhu, C. G. Burd, S. Munro, C. Sander, J. Rine, J. Greenblatt, M. Peter, A. Bretscher, G. Bell, F. P. Roth, G. W. Brown, B. Andrews, H. Bussey and C. Boone (2004): “Global mapping of the yeast genetic interaction network,” Science, 303, 808–813.Google Scholar

  • Wang, C., J. Xuan, L. Chen, P. Zhao, Y. Wang, R. Clarke and E. Hoffman (2008): “Motif-directed network component analysis for regulatory network inference,” BMC Bioinformatics, 9(Suppl. 1), S21.Google Scholar

  • Warshall, S. (1962): “A theorem on Boolean matrices,” J. ACM., 9, 11–12.CrossrefGoogle Scholar

  • Weile, J., K. James, J. Hallinan, S. J. Cockell, P. Lord, A. Wipat and D. Wilkinson (2012): “Bayesian integration of networks without gold standards,” Bioinformatics, 28, 1495–1500.PubMedCrossrefGoogle Scholar

  • Winter, C., G. Kristiansen, S. Kersting, J. Roy, D. Aust, T. Knösel, P. Rümmele, B. Jahnke, V. Hentrich, F. Rückert, M. Niedergethmann, W. Weichert, M. Bahra, H.J. Schlitt, U. Settmacher, H. Friess, M. Büchler, H.-D. Saeger, M. Schroeder, C. Pilarsky, R. Grützmann (2012): “Google goes cancer: improving outcome prediction for cancer patients by network-based ranking of marker genes,” PLoS Comput. Biol., 8, e1002511.Web of ScienceCrossrefGoogle Scholar

  • Yates, P. D. and N. D. Mukhopadhyay (2013): “An inferential framework for biological network hypothesis tests,” BMC Bioinformatics, 14, 94.PubMedCrossrefWeb of ScienceGoogle Scholar

About the article

Corresponding author: Chris J. Oates, Department of Statistics, Zeeman Building, University of Warwick, Gibbet Hill Road, Coventry, CV4 7AL, UK, e-mail:

Published Online: 2014-08-05

Published in Print: 2014-10-01

Citation Information: Statistical Applications in Genetics and Molecular Biology, Volume 13, Issue 5, Pages 611–631, ISSN (Online) 1544-6115, ISSN (Print) 2194-6302, DOI: https://doi.org/10.1515/sagmb-2014-0012.

Export Citation

©2014 by De Gruyter.Get Permission

Supplementary Article Materials

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

Thalia E. Chan, Michael P.H. Stumpf, and Ann C. Babtie
Cell Systems, 2017, Volume 5, Number 3, Page 251
Ann C. Babtie and Michael P. H. Stumpf
Journal of The Royal Society Interface, 2017, Volume 14, Number 133, Page 20170237

Comments (0)

Please log in or register to comment.
Log in