Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Measurement Science Review

The Journal of Institute of Measurement Science of Slovak Academy of Sciences

6 Issues per year


IMPACT FACTOR 2016: 1.344

CiteScore 2016: 1.88

SCImago Journal Rank (SJR) 2016: 0.495
Source Normalized Impact per Paper (SNIP) 2016: 1.419

Open Access
Online
ISSN
1335-8871
See all formats and pricing
More options …
Volume 10, Issue 2

Issues

On the Optimum Architecture of the Biologically Inspired Hierarchical Temporal Memory Model Applied to the Hand-Written Digit Recognition

Svorad Štolc
  • Safety & Security Department, Austrian Institute of Technology, Seibersdorf, Austria
  • Institute of Measurement Science, Slovak Academy of Sciences, Bratislava, Slovakia
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
/ Ivan Bajla
Published Online: 2010-05-14 | DOI: https://doi.org/10.2478/v10048-010-0008-4

On the Optimum Architecture of the Biologically Inspired Hierarchical Temporal Memory Model Applied to the Hand-Written Digit Recognition

In the paper we describe basic functions of the Hierarchical Temporal Memory (HTM) network based on a novel biologically inspired model of the large-scale structure of the mammalian neocortex. The focus of this paper is in a systematic exploration of possibilities how to optimize important controlling parameters of the HTM model applied to the classification of hand-written digits from the USPS database. The statistical properties of this database are analyzed using the permutation test which employs a randomization distribution of the training and testing data. Based on a notion of the homogeneous usage of input image pixels, a methodology of the HTM parameter optimization is proposed. In order to study effects of two substantial parameters of the architecture: the patch size and the overlap in more details, we have restricted ourselves to the single-level HTM networks. A novel method for construction of the training sequences by ordering series of the static images is developed. A novel method for estimation of the parameter maxDist based on the box counting method is proposed. The parameter sigma of the inference Gaussian is optimized on the basis of the maximization of the belief distribution entropy. Both optimization algorithms can be equally applied to the multi-level HTM networks as well. The influences of the parameters transitionMemory and requestedGroupCount on the HTM network performance have been explored. Altogether, we have investigated 2736 different HTM network configurations. The obtained classification accuracy results have been benchmarked with the published results of several conventional classifiers.

Keywords: hierarchical temporal memory (HTM); optimum network architecture; visual pattern recognition; USPS hand-written digits

  • Felleman, D., van Essen, D. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex (1), 1-47.Google Scholar

  • Serre, T., Oliva, A., Poggio, T. (2007). A feedforward architecture accounts for rapid categorization. In: Proc. National Academy of Sciences of the USA, Vol. 15. pp. 6424-6429.Web of ScienceGoogle Scholar

  • Lee, T. S., Mumford, D. (2003). Hierarchical Bayesian inference in visual cortex. Journal of Optical Society of America A 20(7), 1434-1448.Google Scholar

  • Dean, T. (2006). Scalable inference in hierarchical generative models. In: Proc. 9th Int. Symp. on Artificial Intelligence and mathematics. pp. 1-9.Google Scholar

  • Hawkins, J., Blakeslee, S. (2004). On intelligence. Henry Holt and Company, New York.Google Scholar

  • George, D., Hawkins, J. (2009). Towards a mathematical theory of cortical micro-circuits. PLoS Computational Biology 5(10). DOI 10.1371/journal.pcbi.1000532.Web of ScienceCrossrefPubMedGoogle Scholar

  • George, D., Hawkins, J. (2005). Hierarchical Bayesian model of invariant pattern recognition in the visual cortex. In: Proc. Int. Joint Conf. on Neural Networks. Montreal, Canada.Google Scholar

  • Numenta (2007). Zeta1 algorithms reference. Document version 1.0.Google Scholar

  • Dong, J. (2001). Statistical results of human performance on USPS database. Technical report, CEN-PARMI, Concordia University.Google Scholar

  • Dong, J. (2005). HeroSvm 2.1. http://www.cenparmi.concordia.ca/~jdong/HeroSvm.html

  • Thornton, J. R., Gustafsson, T., Blumenstein, M., Hine, T. (2006). Robust character recognition using hierarchical Bayesian network. In: Proc. 19th Australian Joint Conf. on Artificial Intelligence, Hobart, Australia. pp. 1259-1264.Google Scholar

  • Thornton, J. R., Faichney, J., Blumenstein, M., Hine, T. (2008). Character recognition using hierarchical vector quantization and temporal pooling. In: Wobcke, W., Zhang, M. (eds.) Proc 21st Australasian Joint Conf. Artificial Intelligence, Vol. Lecture Notes in Computer Science. pp. 562-572.Google Scholar

  • Bobier, B. (2007). Hand-written digit recognition using Hierarchical Temporal Memory. http://arts.uwaterloo.ca/~cnrglab/?q=system/files/SoftComputingFinalProject.pdf

  • Numenta (2009). Numenta forum: benchmark with USPS handwritten digit dataset. http://www.numenta.com/phpBB2/viewtopic.php?t=224

  • Numenta (2008). Hierarchical temporal memory, concepts, theory, and terminology. Document version 1.8.0.Google Scholar

  • George, D. (2008). How the brain might work: a hierarchical and temporal model for learning and recognition. Ph.D. thesis, Dept. of Electrical Engineering, Stanford University, USA.Google Scholar

  • Numenta (2009). Numenta node algorithms guide, NuPIC 1.7.Google Scholar

  • Johnson, S. T. (1967). Hierarchical clustering schemes. Psychometrika 32, 241-254.PubMedCrossrefGoogle Scholar

  • Numenta (2008). Vision framework guide, NuPIC 1.6.1.Google Scholar

  • Goldberg, D. E. (1989). Genetic algorithms in search, optimization and machine learning. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.Google Scholar

  • Wang, C. H., Srihari, S. N. (1988). A framework for object recognition in a visually complex environment and its application to locating address blocks on mail pieces. Int. Journal of Computer Vision 2(2), 125-151.Google Scholar

  • Dong, J., Krzyzak, A., Suen, C. Y. (2001). Statistical results of human performance on USPS database. Technical report, Centre of Pattern Recognition and Machine Intelligence, Concordia University.Google Scholar

  • Seewald, A. K. (2005). Digits-a dataset for hand-written digit recognition. Technical Report TR-2005-27, OFAI, Wien.Google Scholar

  • Hull, J. J. (1994). A database for hand-written text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence 16, 550-554.Google Scholar

  • LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., Jackel, L. D. (1989). Back-propagation applied to handwritten zip code recognition. Neural Computing 1(4), 541-551.Google Scholar

  • Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science 19(4), 676-685. DOI 10.1214/088342304000000396.CrossrefGoogle Scholar

  • Schroeder, M. R. (1991). Fractals, chaos, power laws: minutes from an infinite paradise. W. H. Freeman, New York.Google Scholar

  • Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal 27, 379-423.Google Scholar

  • Martin, K. J., Hirschberg, D. S. (1996). Small sample statistics for classification error rates II: confidence intervals and significance tests.Google Scholar

About the article


Published Online: 2010-05-14

Published in Print: 2010-01-01


Citation Information: Measurement Science Review, Volume 10, Issue 2, Pages 28–49, ISSN (Online) 1335-8871, DOI: https://doi.org/10.2478/v10048-010-0008-4.

Export Citation

This content is open access.

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

[1]
L Rodriguez-Cobo, J Mirapeix, A Cobo, and JM Lopez-Higuera
Journal of Intelligent Material Systems and Structures, 2015, Volume 26, Number 10, Page 1243
[2]
Luis Rodriguez-Cobo, Ruben Ruiz-Lombera, Olga M. Conde, Jose-Miguel López-Higuera, Adolfo Cobo, and Jesus Mirapeix
Sensors and Actuators A: Physical, 2013, Volume 204, Page 58

Comments (0)

Please log in or register to comment.
Log in