Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Open Computer Science

Editor-in-Chief: van den Broek, Egon

Covered by:
Web of Science - Emerging Sources Citation Index

CiteScore 2018: 0.63
Source Normalized Impact per Paper (SNIP) 2018: 0.604

ICV 2017: 98.90

Open Access
See all formats and pricing
More options …

A novel performance evaluation paradigm for automated video surveillance systems

Chung-Hao Chen / Yi Yao / Andreas Koschan / Mongi Abidi
Published Online: 2011-12-27 | DOI: https://doi.org/10.2478/s13537-011-0030-0


Most existing performance evaluation methods concentrate on defining various metrics over a wide range of conditions and generating standard benchmarking video sequences to examine the effectiveness of a video tracking system. It is a common practice to incorporate a robustness margin or factor into the system/algorithm design. However, these methods, deterministic approaches, often lead to overdesign, thus increasing costs, or underdesign, causing frequent system failures. In order to overcome the aforementioned limitations, we propose an alternative framework to analyze the physics of the failure process via the concept of reliability. In comparison with existing approaches where system performance is evaluated based on a given benchmarking sequence, the advantage of our proposed framework lies in that a unified and statistical index is used to evaluate the performance of an automated video surveillance system independent of input sequences. Meanwhile, based on our proposed framework, the uncertainty problem of a failure process caused by the system’s complexity, imprecise measurements of the relevant physical constants and variables, and the indeterminate nature of future events can be addressed accordingly.

Keywords: performance evaluation; reliability; surveillance systems; and video tracking

  • [1] Bashir F., Porikli F., Performance evaluation of object detection and tracking systems, In: Proceedings of the 9th IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance (VS-PETS 06), New York, NY, USA, 2006 Google Scholar

  • [2] Collins R.T., Lipton A.J., Kanade, T., Introduction to the special section on video surveillance, IEEE T. Pattern Anal., 22(8), 745–746, 2000 http://dx.doi.org/10.1109/TPAMI.2000.868676CrossrefGoogle Scholar

  • [3] Calderara S., Prati A., Vezzani R., Cucchiara R., Consistent labeling for multi-camera object tracking, In: The 13th International Conference on Image Analysis and Processing, Roli, F., Vitulano, S. (Eds.), Springer, LNCS 3617, 1206–1214, 2005 Google Scholar

  • [4] Chau D.P., Bremond F., Thonnat M. Online evaluation of tracking algorithm performance, In: The Int. Conf. on Imaging for Crime Detection and Prevention, 1–6, 2009 Google Scholar

  • [5] Cui Y., Samarasekera S., Huang Q., Greiffenhagen M., Indoor monitoring via the collaboration between a peripheral sensor and a foveal sensor, In: IEEE Workshop on Visual Surveillance, 2–9, 1998 Google Scholar

  • [6] Doermann D. Mihalcik D., Tools and techniques for video performance evaluation, In: The 15th Int. Conf. on Pattern Recognition, vol. 4, 167–170, 2000 http://dx.doi.org/10.1109/ICPR.2000.902888CrossrefGoogle Scholar

  • [7] Dodson B. Nolan D. Reliability engineering handbook, CRC Press, 1999 Google Scholar

  • [8] Dai Y.-S., Xie M., Log Q., Ng S.-H., Uncertainty analysis in software reliability modeling by Bayesian approach with maximum-entropy principle, IEEE T. Software Eng., 33(11), 781–795, 2007 http://dx.doi.org/10.1109/TSE.2007.70739Web of ScienceCrossrefGoogle Scholar

  • [9] Ebeling C.E., An introduction to reliability and maintainability engineering, McGraw-Hill, 1997 Google Scholar

  • [10] Erdem C., Tekalp A., Sankur B., Metrics for performance evaluation of video object segmentation and tracking without ground-truth, IEEE Image Proc., 2, 69–72, 2001 Google Scholar

  • [11] Jaynes C., Webb S., Steele R. M., Xiong Q., An open development environment for evaluation of video surveillance systems, In: The 3rd Int. Workshop on Performance Evaluation of Tracking and Surveillance, 2002 Google Scholar

  • [12] Kapur J., Maximum-entropy models in science and engineering, John Wiley & Sons, 1989 Google Scholar

  • [13] Kasturi R., Goldgof D., Soundararajan P., Manohar V., Garofolo J., Bowers R., Boonstra M., Korzhova V., Zhang J., Framework for performance evaluation of face, text, and vehicle detection and tracking in video: data, metrics, and protocol, IEEE T. Pattern Anal., 31(2), 319–336, 2009 http://dx.doi.org/10.1109/TPAMI.2008.57Web of ScienceCrossrefGoogle Scholar

  • [14] Lazarevic-McManus N., Renno J., Jones G. A., Performance evaluation in visual surveillance using the F-measure, In: The 4th ACM Int. Workshop on Video Surveillance and Sensor-Networks, 45–52, 2006 Google Scholar

  • [15] List, T., Bins, J., Vazquez, J., Fisher, R.B., Performance evaluating the evaluator, In: Proc. 2nd Joint IEEE Int. Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, (VS-PETS), Beijing, 129–136, 2005 Google Scholar

  • [16] Lei R., Xu L.-Q., Real-time outdoor video surveillance with robust foreground extraction and object tracking via multi-state transition management, Pattern Recogn. Lett., 27(15), 1816–1825, 2006 http://dx.doi.org/10.1016/j.patrec.2006.02.017CrossrefGoogle Scholar

  • [17] Nawaz T., Cavallaro, A., PFT: A protocol for evaluating video trackers, IEEE Image Proc. 2011 Google Scholar

  • [18] Schlögl T., Beleznai C., Winter M., Bischof H., Performance evaluation metrics for motion detection and tracking, In: The 17th Int. Conf. on Pattern Recognition, 4, 519–522, 2004 Google Scholar

  • [19] Pan P., Porikli F., Schonfeld D., A new method for tracking performance evaluation based on a reflective model and perturbation analysis, IEEE ICASSP, 3529–3532, 2009 Google Scholar

  • [20] Perera A. G. A., Hooqs A., Srnivas C., Brooksby G., Wensheng H., Evaluation of algorithms for tracking multiple objects in video, In: The 35th IEEE Applied Imagery and pattern Recognition Workshop, 35–35, 2006 Google Scholar

  • [21] Popoola J., Amer A., Performance evaluation for tracking algorithms using object labels, Int. Conf. Acoust. Spee., 733–736, 2008 Google Scholar

  • [22] Wackerly D.D., Mendenhall III W., Scheaffer R.L., Mathematical statistics with applications, 2nd edition, Duxbury Press, 2002 Google Scholar

  • [23] Yilmaz A., Javed O., Shah M., Object tracking: a survey, ACM Comput. Surv. 38(4), 13, 2006 http://dx.doi.org/10.1145/1177352.1177355CrossrefGoogle Scholar

  • [24] Zhang M., Chen K., Guo Y.Y., Online parameter based Kalman filter precision evaluation method for video tracking, In: IEEE Int. Conference on Multimedia Technology, 598–601, 2011 Google Scholar

About the article

Published Online: 2011-12-27

Published in Print: 2011-12-01

Citation Information: Open Computer Science, Volume 1, Issue 4, Pages 430–441, ISSN (Online) 2299-1093, DOI: https://doi.org/10.2478/s13537-011-0030-0.

Export Citation

© 2011 Versita Warsaw. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License. BY-NC-ND 3.0

Comments (0)

Please log in or register to comment.
Log in