Jump to ContentJump to Main Navigation
Show Summary Details
In This Section

Open Computer Science

Editor-in-Chief: van den Broek, Egon

1 Issue per year

Open Access
Online
ISSN
2299-1093
See all formats and pricing
In This Section

Pruning algorithms of neural networks — a comparative study

M. Augasta
  • Department of Computer Applications, Sarah Tucker College, Tirunelveli, 627007, TN, India
  • Email:
/ T. Kathirvalavakumar
  • Department of Computer Science, V.H.N.S.N College, ViruduNagar, 626001, TN, India
  • Email:
Published Online: 2013-09-25 | DOI: https://doi.org/10.2478/s13537-013-0109-x

Abstract

The neural network with optimal architecture speeds up the learning process and generalizes the problem well for further knowledge extraction. As a result researchers have developed various techniques for pruning the neural networks. This paper provides a survey of existing pruning techniques that optimize the architecture of neural networks and discusses their advantages and limitations. Also the paper evaluates the effectiveness of various pruning techniques by comparing the performance of some traditional and recent pruning algorithms based on sensitivity analysis, mutual information and significance on four real datasets namely Iris, Wisconsin breast cancer, Hepatitis Domain and Pima Indian Diabetes.

Keywords: input and hidden neurons pruning; optimization techniques; classification; feedforward neural networks; data mining

  • [1] P. M. Atkinson, A. R. L. Tatnall, Neural networks in remote sensing, Int. J. Remote Sens. 18(4), 699, 1997 http://dx.doi.org/10.1080/014311697218700CrossrefGoogle Scholar

  • [2] A. Fangju, A New Pruning Algorithm for Feedforward Neural Networks, Fourth International Workshop on Advanced Computational Intelligence, IEEE Conference Publication, Wuhan, Hubei, China 19–21 October 2011, 286–289 Google Scholar

  • [3] A. Yoan, A. Sorjamaa, P. Bas, O. Simula, C. Jutten, A. Lendasse, 3. OP-ELM: optimally pruned extreme learning machine, IEEE Trans. Neural Networks 21(1), 158–162, 2010 http://dx.doi.org/10.1109/TNN.2009.2036259Web of ScienceCrossrefGoogle Scholar

  • [4] S. Ahmmed, K. Abdullah-Al-Mamun, M. Islam, A novel algorithm for designing three layered artificial neural networks, Int. J. Soft. Comput. 2(3), 450–458, 2007 Google Scholar

  • [5] O. Aran, O. T. Yildiz, E. Alpaydin, An incremental framework based on cross validation for estimating the architecture of a multilayer perceptron, Int. J. Pttern. Recogn. Artif. Intell. 23(2), 159–190, 2009 http://dx.doi.org/10.1142/S0218001409007132Web of ScienceCrossrefGoogle Scholar

  • [6] J. Xua, D. W. C. Hob, A new training and pruning algorithm based on node dependence and Jacobian rank deficiency, Neurocomputing 70, 544–558, 2006 http://dx.doi.org/10.1016/j.neucom.2005.11.005CrossrefGoogle Scholar

  • [7] B. Choi, J. HongLee, D.-H. Kim, Solving local minima problem with large number of hidden nodes on two layered feedforward artificial neural networks, Neurocomputing 71, 3640–3643, 2008 http://dx.doi.org/10.1016/j.neucom.2008.04.004CrossrefGoogle Scholar

  • [8] D. Sabo, X.-H. Yu, A new pruning algorithm for neural network dimension analysis, IJCNN 2008, IEEE World Congress on Computational Intelligence, In Proc. of IEEE Int. Joint Conference on Neural Networks, Hong Kong, 1–8 June 2008, 3313–3318 Google Scholar

  • [9] R. Reed, Pruning algorithms a survey, IEEE T. Neural Networ. 4(5), 740–747, 1993 http://dx.doi.org/10.1109/72.248452CrossrefGoogle Scholar

  • [10] R. Setiono, H. Liu, Understanding Neural Networks via Rule Extraction, In: Proc. of 14th International Joint Conference on Artificial Intelligence, Montreal, Canada, August 20–25 1995, 480–485 Google Scholar

  • [11] M. D. Emmerson, R. I. Damper, Determining and improving the fault tolerance of multi layer perceptrons in a pattern-recognition application, IEEE T. Neural Networ. 4, 788–793, 1993 http://dx.doi.org/10.1109/72.248456CrossrefGoogle Scholar

  • [12] J. M. Zurada, Introduction to Artificial Neural Systems (Jaisco Publishing House, Mumbai, 2002) Google Scholar

  • [13] R. Setiono, B. Baesens, C. Mues, A note on knowledge discovery using neural networks and its application to credit card screening, Eur. J. Oper. Res. 192(1), 326–332, 2008 http://dx.doi.org/10.1016/j.ejor.2007.09.022Web of ScienceCrossrefGoogle Scholar

  • [14] M. G. Augasta, T. Kathirvalavakumar, Reverse Engineering the Neural Networks for Rule Extraction in Classification Problems, Neural Process. Lett. 35, 131–150, 2012 http://dx.doi.org/10.1007/s11063-011-9207-8CrossrefWeb of ScienceGoogle Scholar

  • [15] A. P. Engelbrecht, A new pruning heuristic based on variance analysis of sensitivity information, IEEE T. Neural Networ. 12(6), 1386–1399, 2001 http://dx.doi.org/10.1109/72.963775CrossrefGoogle Scholar

  • [16] T. Q. Huynh, R. Setiono, Effective neural network pruning using cross validation, In: Proc. of IEEE Int. Joint Conference on Neural Networks 2, Montreal, 31 July–4 August 2005, 972–977 Google Scholar

  • [17] G. Castellano, A. M. Fanelli, M. Pelillo, An Iterative Pruning algoritm for feedforward neural networks, IEEE T Neural Networ. 8(3), 519–530, 1997 http://dx.doi.org/10.1109/72.572092CrossrefGoogle Scholar

  • [18] S. Marsland, S. U. Nehmzow, J. Shapiro, A self organizing network that grows when required, Neural Networ. 15(809), 1041–1058, 2002 http://dx.doi.org/10.1016/S0893-6080(02)00078-3CrossrefGoogle Scholar

  • [19] R. Zhang, Y. Lan, G. B. Huang, Z. B. Xu, Universal approximation of extreme learning machine with adaptive growth of hidden nodes, IEEE T. Neural Networ. Learn. Syst. 23(2), 365–371, 2012 http://dx.doi.org/10.1109/TNNLS.2011.2178124CrossrefGoogle Scholar

  • [20] G. B. Huang, L. Chen, 20. Enhanced random search based incremental extreme learning machine, Neuro Comput. 71(16–18), 3460–3468, 2008 Google Scholar

  • [21] A. B. Nielsen, L. K. Hansen, Structure learning by pruning in independent component analysis, Neuro Comput. 71(10–12), 2281–2290, 2008 Google Scholar

  • [22] D. Sabo, X.-H. Yu, Neural network dimension selection for dynamical system identification, In: Proc. of 17th IEEE International Conference on Control Applications, San Antonio, TX, 3–5 September 2008, 972, 977 Google Scholar

  • [23] S. C. Huang, Y. F. Huang, Bounds on the number of hidden neurons in multilayer perceptrons, IEEE T. Neural Networ. 2, 47–55, 1991 http://dx.doi.org/10.1109/72.80290CrossrefGoogle Scholar

  • [24] H.-G. Han, J.-F. Qiao, A structure optimisation algorithm for feedforward neuralnetwork construction, Neurocomputing 99, 347–357, 2013 http://dx.doi.org/10.1016/j.neucom.2012.07.023Web of ScienceCrossrefGoogle Scholar

  • [25] P. L. Narasimhaa, W. H. Delashmitb, M. T. Manrya, J. Lic, F. Maldonado, An integrated growing-pruning method for feedforward network training, Neurocomputing 71, 2831–2847, 2008 http://dx.doi.org/10.1016/j.neucom.2007.08.026Web of ScienceCrossrefGoogle Scholar

  • [26] A. B. Nielsen, L. K. Hansen, Structure learning by pruning in independent component analysis, Neurocomputing, 71(10–12), 2281–2290, 2008 http://dx.doi.org/10.1016/j.neucom.2007.09.016Web of ScienceCrossrefGoogle Scholar

  • [27] M. Attik, L. Bougrain, F. Alexandra, Neural Network topology optimization, In: Proceedings of ICANN’05, Lecture Notes in Computer Science, Vol. 3697, 5th International Conference, Warsaw, Poland, 11–15 September, 2005 (Springer, Berlin, Heidelberg, 2005) 53–58 Google Scholar

  • [28] Q. Jun-fei, Z. Ying, H. Hong-gui, Fast unit pruning algorithm for feed-forward neural network design, App. Math. Comput. 205(2), 662–667, 2008 Web of ScienceGoogle Scholar

  • [29] N. Fnaiech, S. Abid, F. Fnaiech, M. Cheriet, A modified version of a formal pruning algorithm based on local relative variance analysis, First International IEEE Symposium on Control, Communications and Signal Processing, Hammamet, Tunisia, 21–24 March, 2004, 849, 852 http://dx.doi.org/10.1109/ISCCSP.2004.1296579CrossrefGoogle Scholar

  • [30] R. Setiono, A penalty function approach for pruning feedforward neural networks, Neural Comput. 9(1), 185–204, 1997 http://dx.doi.org/10.1162/neco.1997.9.1.185CrossrefGoogle Scholar

  • [31] W. Wan, S. Mabu, K. Shimada, K. Hirasawa, Enhancing the generalization ability of neural networks through controlling the hidden layers, J. Hu, App. Soft Comput. 9, 404–414, 2009 http://dx.doi.org/10.1016/j.asoc.2008.01.013CrossrefWeb of ScienceGoogle Scholar

  • [32] M. Hagiwara, A simple and effective method for removal of hidden units and weights, Neurocomputing, 6, 207–218, 1994 http://dx.doi.org/10.1016/0925-2312(94)90055-8CrossrefGoogle Scholar

  • [33] J. Sietsma, Dow RJF, Neural net pruning: why and how, In: Proc. of the IEEE International Conference on Neural Networks, Vol. 1, San Diego, CA, USA, 24–27 July 1988, 325–333 http://dx.doi.org/10.1109/ICNN.1988.23864CrossrefGoogle Scholar

  • [34] H.-J. Xing, B.-G. Hu, Two phase construction of multilayer perceptrons using Information Theory, IEEE T. Neural Networ. 20(4), 715–721, 2009 http://dx.doi.org/10.1109/TNN.2008.2005604Web of ScienceCrossrefGoogle Scholar

  • [35] Z. Zhang, J. Qiao, A Node Pruning Algorithm for Feedforward Neural Network Based on Neural Complexity, In: Int. Conf. on Intelligent Control and Information Processing, Dalian, 13–15 August 2010, 406–410 Google Scholar

  • [36] D. Whitley, C. Bogart, The evolution of connectivity: Pruning neural networks using genetic algorithms, In: Int. Joint Conf. Neural Networks, 1 (IEE Press, Washington DC, 1990) 134–137 Google Scholar

  • [37] P. G. Benardos, G.-C. Vosniakos, Optimizing feedforward artificial neural network architecture, Eng. App. Artif. Intelligence, 20, 365–382, 2007 http://dx.doi.org/10.1016/j.engappai.2006.06.005Web of ScienceCrossrefGoogle Scholar

  • [38] X. Zeng, D. S. Yeung, Hidden neuron pruning of multilayer perceptrons using a quantified sensitivity measure, Neuro Comput. 69, 825–837, 2006 Google Scholar

  • [39] P. Lauret, E. Fock, T. A. Mara, A Node Pruning Algorithm Based on a Fourier Amplitude Sensitivity Test Method, IEEE T. Neural Networ. 17(2), 273–293, 2006 http://dx.doi.org/10.1109/TNN.2006.871707CrossrefGoogle Scholar

  • [40] Y. Le Cun, J. S. Denker, S. A. Solla, In. D. S. Touretzky (Ed.), Optimal brain damage, Advances in neural information processing systems (Morgan Kaufmann, San Mateo, 1990) 2, 598–605 Google Scholar

  • [41] B. Hassibi, D. G. Stork, G. J. Wolf, Optimal brain surgeon and general network pruning, In: Proc. of IEEE ICNN’93, 1, WDS’08 Proceedings of Contributed Papers, Part I, 2008, 293–299 Google Scholar

  • [42] W. U. Jian-yu, H. E. Xiao-rong, DOBD Algorithm for Training Neural Network, Part I. Method, Chinese J. Process Eng. 2(2), 172–176, 2002 Google Scholar

  • [43] P. V. S. Ponnapallii, K. C. Ho, M. Thomson, A formal selection and pruning algorithm for feedforward artificial neural network optimization, IEEE T. Neural Networ., 10(4), 964–968, 1999 http://dx.doi.org/10.1109/72.774273CrossrefGoogle Scholar

  • [44] L. M. Belue, K. W. Bauer, Determining input features for multilayer perceptrons, Neurocomputing 7, 111–121, 1995 http://dx.doi.org/10.1016/0925-2312(94)E0053-TCrossrefGoogle Scholar

  • [45] G. Augasta, T. Kathirvalavakumar, A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems, Neural Process. Lett. 34(3), 241–258, 2011 http://dx.doi.org/10.1007/s11063-011-9196-7Web of ScienceCrossrefGoogle Scholar

  • [46] T. Ragg, H. Braun, H. Landsberg, A comparative study of neural network optimization Techniques, In 13th International Conf. on Machine Learning, Norwich, UK, 2–4 April, 1997, Artificial Nets and Genetic Algorithms (Springer, 1997) 341–345 Google Scholar

About the article

Published Online: 2013-09-25

Published in Print: 2013-09-01


Citation Information: Open Computer Science, ISSN (Online) 2299-1093, DOI: https://doi.org/10.2478/s13537-013-0109-x.

Export Citation

© 2013 Versita Warsaw. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License. BY-NC-ND 3.0

Comments (0)

Please log in or register to comment.
Log in