Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Paladyn, Journal of Behavioral Robotics

Editor-in-Chief: Schöner, Gregor

Covered by SCOPUS

CiteScore 2018: 2.17

SCImago Journal Rank (SJR) 2018: 0.336
Source Normalized Impact per Paper (SNIP) 2018: 1.707

ICV 2017: 99.90

Open Access
See all formats and pricing
More options …

Open-source benchmarking for learned reaching motion generation in robotics

A. Lemme / Y. Meirovitch / M. Khansari-Zadeh / T. Flash / A. Billard / J. J. Steil
Published Online: 2015-03-10 | DOI: https://doi.org/10.1515/pjbr-2015-0002


This paper introduces a benchmark framework to evaluate the performance of reaching motion generation approaches that learn from demonstrated examples. The system implements ten different performance measures for typical generalization tasks in robotics using open source MATLAB software. Systematic comparisons are based on a default training data set of human motions, which specify the respective ground truth. In technical terms, an evaluated motion generation method needs to compute velocities, given a state provided by the simulation system. This however is agnostic to how this is done by the method or how the methods learns from the provided demonstrations. The framework focuses on robustness, which is tested statistically by sampling from a set of perturbation scenarios. These perturbations interfere with motion generation and challenge its generalization ability. The benchmark thus helps to identify the strengths and weaknesses of competing approaches, while allowing the user the opportunity to configure the weightings between different measures.

Keywords : benchmarking; standardized comparisons; human-like motions; reaching motions; movement primitive; dynamical systems; learning from demonstrations; programming by demonstrations


  • [1] B. Adams, C.L. Breazeal, R. Brooks, and B. Scassellati. Humanoid robots: a new kind of tool. IEEE Intell. Syst. Control lett., 15(4):25–31, 2000. Google Scholar

  • [2] Alin Albu-Schäffer, Sami Haddadin, Ch Ott, Andreas Stemmer, ThomasWimböck, and Gerd Hirzinger. The dlr lightweight robot: design and control concepts for robots in human environments. Industrial Robot: An International Journal, 34(5):376–385, 2007. Google Scholar

  • [3] E. Oztop, D. Franklin, T. Chaminade, and G. Cheng. Human– humanoid interaction: Is a humanoid robot perceived as a human? Humanoid Robotics, 2(04):537–559, 2005. Google Scholar

  • [4] T. Chaminade, D. Franklin, E. Oztop, and G. Cheng. Motor interference between humans and humanoid robots: Effect of biological and artificial motion. In Proc. of Int. Conf. on Development and Learning, pages 96–101. IEEE, 2005. Google Scholar

  • [5] Aleš Ude, Christopher G. Atkeson, and Marcia Riley. Programming full-body movements for humanoid robots by observation. Robotics and Autonomous Systems, 47(2-3):93–108, 2004. CrossrefGoogle Scholar

  • [6] Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. Robot Programming by Demonstration, chapter 59, pages 1371–1394. Springer, 2008. Google Scholar

  • [7] Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469–483, 2009. CrossrefGoogle Scholar

  • [8] D. Bailey and J. Barton. The NAS kernel benchmark program. National Aeronautics and Space Administration, Ames Research Center, 1985. Google Scholar

  • [9] K. Gaj, E. Homsirikamol, and M. Rogawski. Fair and comprehensive methodology for comparing hardware performance of fourteen round two SHA-3 candidates using FPGAs. In Cryptographic Hardware and Embedded Systems (CHES), pages 264– 278. Springer, 2010. Google Scholar

  • [10] E. Dolan and J. More. Benchmarking optimization software with performance profiles. Mathematical programming, 91(2):201– 213, 2002. CrossrefGoogle Scholar

  • [11] A. Paraschos, G. Neumann, and J. Peters. A probabilistic approach to robot trajectory generation. In Proc. of Int. Conf. on Humanoid Robots (Humanoids), pages 477–483, 2013. Google Scholar

  • [12] S. Calinon, T. Alizadeh, and D. G. Caldwell. On improving the extrapolation capability of task-parameterized movement models. In Proc. of Int. Conf on Intelligent Robots and Systems (IROS), pages 610–616. IEEE, 2013. Google Scholar

  • [13] S.M. Khansari-Zadeh and A. Billard. Learning stable nonlinear dynamical systems with gaussian mixture models. Transactions on Robotics, 27(5):943–957, 2011. Google Scholar

  • [14] A. Lemme, Neumann K., F. R. Reinhart, and J. J. Steil. Neural learning of vector fields for encoding stable dynamical systems. Neurocomputing, 141(0):3–14, 2014. Google Scholar

  • [15] K. Neumann, Lemme A., and J. J. Steil. Neural learning of stable dynamical systems based on data-driven Lyapunov candidates. In Proc. of Int. Conf. Intelligent Robots and Systems (IROS), pages 1216–1222. IEEE, 2013. Google Scholar

  • [16] A. Ude, A. Gams, T. Asfour, and J. Morimoto. Task-specific generalization of discrete and periodic dynamic movement primitives. Transactions on Robotics, 26(5):800–815, 2010. Google Scholar

  • [17] H. Hoffmann, P. Pastor, Dae-Hyung Park, and S. Schaal. Biologically-inspired dynamical systems for movement generation: Automatic real-time goal adaptation and obstacle avoidance. In Proc. of Int. Conf. on Robotics and Automation (ICRA), pages 2587–2592, 2009. Google Scholar

  • [18] P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal. Learning and generalization of motor skills by learning from demonstration. In Proc. of Int. Conf. on Robotics and Automation (ICRA), pages 763–768. IEEE, 2009. Google Scholar

  • [19] S. Schaal, J. Peters, J. Nakanishi, and A. Ijspeert. Learning movement primitives. In Robotics Research, pages 561–572. Springer, 2005. Google Scholar

  • [20] F. Lacquaniti, C. Terzuolo, and P. Viviani. The law relating kinematic and figural aspects of drawing movements. Acta Psychologica, 54:115–130, 1983. Google Scholar

  • [21] P. Viviani and M. Cenzato. Segmentation and coupling in complex movements. Journal Experimental Psychology Humam Perception and Performence, 11(6):828–845, 1985. Google Scholar

  • [22] N. Hogan. An organizing principle for a class of voluntary movements. Journal of Neuroscience, 4(11):2745, 1984. Google Scholar

  • [23] T. Flash and N. Hogan. The coordination of arm movements - an experimentally confirmed mathematical-model. Journal of Neuroscience, 5(7):1688–1703, 1985. Google Scholar

  • [24] E. Todorov and M. I. Jordan. Smoothness maximization along a predefined path accurately predicts the speed profiles of complex arm movements. J. Neurophysiol, 80:696–714, 1998. Google Scholar

  • [25] T. Flash, Y. Meirovitch, and A. Barliya. Models of human movement: trajectory planning and inverse kinematics studies. Robotics and Autonomous Systems, 61(4):330–339, 2013. CrossrefGoogle Scholar

  • [26] S.M. Khansari-Zadeh. Benchmark data. http://www.amarsiproject. eu/open-source, 2012. Google Scholar

  • [Online; accessed 17-October- 2014]. Google Scholar

  • [27] S.-M. Khansari-Zadeh and Aude Billard. BM: An iterative algorithm to learn stable non-linear dynamical systems with gaussian mixture models. In Proc. of Int. Conf. on Robotics and Automation (ICRA), pages 2381–2388, 2010. Google Scholar

  • [28] S.M. Khansari-Zadeh and A. Billard. Imitation learning of globally stable non-linear point-to-point robot motions using nonlinear programming. In Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), pages 2676–2683, 2010. Google Scholar

  • [29] S. M. Khansari-Zadeh. A Dynamical System-based Approach to Modeling Stable Robot Control Policies via Imitation Learning. PhD thesis, EPFL, 2012. Google Scholar

  • [30] S.M. Khansari-Zadeh and A. Billard. Learning control lyapunov function to ensure stability of dynamical system-based robot reaching motions robotics and autonomous systems. Robotics and Autonomous Systems, 62(6):752–765, 2014. CrossrefGoogle Scholar

  • [31] J. Gómez, D. Alvarez, S. Garrido, and L. Moreno. Kinesthetic teaching via fast marching square. In Proc. of Int. Conf. on Intelligent Robots and Systems (IROS), pages 1305–1310. IEEE, 2012. Google Scholar

  • [32] R. Shadmehr and F. Mussa-Ivaldi. Adaptive representation of dynamics during learning of a motor task. Neuroscience, 14(5):3208–3224, 1994. Google Scholar

  • [33] F. Gandolfo, F.Mussa-Ivaldi, and E. Bizzi. Motor learning by field approximation. National Academy of Sciences, 93(9):3843– 3846, 1996. Google Scholar

  • [34] M. Conditt, F. Gandolfo, and F. Mussa-Ivaldi. The motor system does not learn the dynamics of the arm by rote memorization of past experience. Neurophysiology, 78(1):554–560, 1997. Google Scholar

  • [35] A Karniel and F. Mussa-Ivaldi. Does the motor control system use multiple models and context switching to cope with a variable environment? Experimental Brain Research, 143(4):520– 524, 2002. CrossrefGoogle Scholar

  • [36] D. Sternad and S. Schaal. Segmentation of endpoint trajectories does not imply segmented control. Experimental Brain Research, 124(1):118–136, 1999. CrossrefGoogle Scholar

  • [37] F. Pollick, U. Maoz, A. Handzel, P. Giblin, G. Sapiro, and T. Flash. Three-dimensional arm movements at constant equiaflne speed. Cortex, 45(3):325–339, 2009. CrossrefGoogle Scholar

  • [38] D. Bennequin, R. Fuchs, A. Berthoz, and T. Flash. Movement timing and invariance arise from several geometries. PLoS computational biology, 5(7):e1000426, 2009. CrossrefGoogle Scholar

  • [39] K. Lashley. The problem of serial order in psychology. Cerebral mechanisms in behavior. New York: Wiley, 1951. Google Scholar

  • [40] N. Bernstein. The Co-ordination and Regulation of Movements. Pergamon Press, Oxford, 1967. Google Scholar

  • [41] W. Abend, E. Bizzi, and P. Morasso. Human arm trajectory formation. Exp Brain Res, 105:331–348, 1982. Google Scholar

  • [42] T. Flash. Organizing principles underlying the formation of hand trajectories. Doctoral dissertation, Massachusetts Institute of Technology, Cambridge, MA, 1983. Google Scholar

  • [43] C. M. Harris and D. M. Wolpert. Signal-dependent noise determines motor planning. Nature, 394:780–784, 1998. Google Scholar

  • [44] Bizzi E. Mussa-Ivaladi F.A. Motor learning through the combination of primitives. Philosophical Transactions of the Royal Society of London Series B-Biological Sciences, 355:1755–1759, 2000. Google Scholar

  • [45] E. Bizzi, M.C. Tresch, P. Saltiel, and A. d Avella. New perspectives on spinal motor systems. Nature Reviews Neuroscience, 1(2):101–108, 2000. CrossrefGoogle Scholar

  • [46] T. Flash and B. Hochner. Motor primitives in vertebrates and invertebrates. Current Opinion in Neurobiology, 15(6):660 – 666, 2005. Motor sytems / Neurobiology of behaviour. CrossrefGoogle Scholar

  • [47] P. Viviani and C. deSperati. The relationsheep between curvature and velocity in two dimensional smooth pursuit eye movement. The Journal of Neuroscience, 17:3932–3945, 1997. Google Scholar

  • [48] D. Endres, Y. Meirovitch, T. Flash, and M. Giese. Segmenting sign language into motor primitives with Bayesian binning. Frontiers in computational neuroscience, 7, 2013. Google Scholar

  • [49] F. Polyakov, E. Stark, R. Drori, M. Abeles, and T. Flash. Parabolic movement primitives and cortical states: merging optimality with geometric invariance. Biological Cybernetics, 100(2):159– 184, 2009. CrossrefGoogle Scholar

  • [50] M.S. Khansari, A. Lemme, Y. Meirovitch, B. Schrauwen, M. A. Giese, A.J. Ijspeert, A. Billard, and J.J. Steil. Workshop on benchmarking of state-of-the-art algorithms in generating human-like robot reaching motions. In Humanoids. IEEE, 2013. Google Scholar

  • [51] A. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25(2):328–373, 2013. CrossrefGoogle Scholar

About the article

Received: 2014-08-25

Accepted: 2014-12-11

Published Online: 2015-03-10

Citation Information: Paladyn, Journal of Behavioral Robotics, Volume 6, Issue 1, ISSN (Online) 2081-4836, DOI: https://doi.org/10.1515/pjbr-2015-0002.

Export Citation

© 2015 A. Lemme et al.. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License. BY-NC-ND 3.0

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

Hongmin Wu, Yisheng Guan, and Juan Rojas
Applied Sciences, 2019, Volume 9, Number 6, Page 1072
Mina Alibeigi, Majid Nili Ahmadabadi, and Babak Nadjar Araabi
IEEE Transactions on Robotics, 2017, Volume 33, Number 1, Page 153
Patrick Göttsch, Robin Olschewski, and Herbert Werner
IFAC-PapersOnLine, 2017, Volume 50, Number 1, Page 11459
René Felix Reinhart
Autonomous Robots, 2017, Volume 41, Number 7, Page 1521
Yaron Meirovitch, Daniel Bennequin, and Tamar Flash
IEEE Transactions on Robotics, 2016, Volume 32, Number 4, Page 837

Comments (0)

Please log in or register to comment.
Log in