The human hand is an extremely articulated system with up to 24 degrees of freedom (DoF) even for a simplified model . The accurate tracking of the hand and finger pose becomes more and more important for animation, virtual reality, sports, and also rehabilitation.
Current measurement systems are primarily based on instrumented gloves with resistive, magnetic, or optical sensing methods . Resistive sensors measure the change in electrical resistance depending on the joint angle(s). The sensors have to be mounted across the finger joints, which requires an accurate placement. Each joint requires a separate sensor. Otherwise, the sum of all joint angles underneath the sensor will be measured. This is why the correct sensor attachment is crucial and why the sensors are often integrated into gloves, which simplify the sensor attachment. A specific calibration procedure is still necessary, depending on the number of sensors and the required accuracy. This might be a problem if the hand function is impaired, prohibiting a patient to execute the calibration movements. Optical data gloves based on fiber optics also measure the joint angle sum and therefore have the same drawbacks as resistive data gloves, although a higher accuracy is achieved .
Putting on gloves can be very difficult, time consuming, and frustrating if the hand function is impaired. In a clinical environment, cleaning or disinfecting the instrumented glove is also a major issue and may hinder the use of data gloves in this field. Data gloves must have a good fit to minimize the measurement errors, which is why universally sized gloves are not feasible.
Another class of optical systems are camera-based tracking systems. These systems use one or more cameras to track the finger segments, which are often equipped with optical markers to enhance accuracy and robustness. The major disadvantages are the high price, the system complexity, and the dependency on a known camera alignment, which makes this kind of system unsuitable for daily life situations, or rehabilitation. Marker-less systems with one or two cameras are more affordable, but the accuracy and robustness e.g. against overlapping finger segments is severely reduced.
Recently, inertial measurement unit (IMU) based tracking systems have been suggested by Kortier et al. . The finger segments are instrumented with IMUs which measure the acceleration, rate of turn, and in some cases also the magnetic field in three dimensions. Kortier et al. employ an extended Kalman filter to estimate the hand and finger kinematics using assumptions on the kinematic chain of the finger segments.
We present a similar system intended as an integral part of a functional electrical stimulation (FES) based closed-loop neuroprosthesis for grasping . We employ a modular approach in which inertial measurements (i.e. acceleration, rate of rotation, and magnetic field) are obtained for every finger segment separately. For each segment, we perform sensor fusion of these measurements to calculate the segment orientation directly, i.e. without any assumption on the kinematic chain of the finger segments, and then combine the orientations to calculate joint angles.
2 System design
The primary use case of the presented sensor system is the hand and finger assessment and the closed-loop application of FES during rehabilitation of stroke survivors. The main goals of the sensor system are therefore:
accurate tracking of the hand and finger pose,
flexibility regarding the sensor setup (number of tracked fingers),
and convenient attachment of the sensors to an impaired hand.
This translates into the modular design depicted in Figures 1 and 2. The system is not integrated into a glove but consists of up to five sensor strips, a base unit, and one wireless sensor for the forearm. Only the base unit, which is placed on the back of the hand, is mandatory. All other parts can be added depending on the requirements.
The base unit collects the sensor raw data and sends it via USB to the computer. Orientation estimation algorithms on the computer use this raw data to calculate the ISB-conform joint angles , which can be used in open- or closed-loop applications.
2.1 Base unit
The base unit is a custom printed circuit board (PCB) design with one 32 bit ARM Cortex-M4 microcontroller unit (MCU), power supply, high side switches and connectors for the five sensor strips, and one 9 D inertial sensor (MPU9250, InvenSense Inc., San Jose, California, USA). USB 2.0 (full speed; 12 Mbit), which is directly supported by the MCU, is used to communicate with the computer program and to send the IMU raw data to the computer.
The inertial sensor MPU9250 was chosen because of the fully integrated 9 D measurement capabilities, the very small footprint of only 3 × 3 mm, and the support for the Serial Peripheral Interface bus (SPI). The integrated circuit (IC) combines a 3 D accelerometer (ACC), a 3 D gyroscope (GYR), and a 3 D magnetometer (MAG), with one common communication interface. SPI was chosen because of the much higher data transfer rates compared to the Inter-Integrated Circuit bus (I2C), which is usually used with the this type of sensor. The MPU9250 IC is susceptible to power variations, e.g. when connecting a sensor strip to the base unit. Therefore, high side switches are used to enable and disable the sensor strips power supply and to reset the sensors if necessary.
2.2 Sensor strip
Each sensor strip consists of three 9 D inertial sensors (MPU9250), 512 kBit SPI flash memory, and one connector to the base unit. All components are connected by a 19 cm long flexible PCB depicted in Figure 3. The length of the sensor strips was chosen to support different hand sizes and to deal with the length variation of the straight and bent finger. A maximum width of only 6.5 mm reduces the interference with the finger movements to a minimum.
A unique strip ID, as well as strip-specific IMU calibration data can be stored on the flash memory. The strip ID is used to detect which sensor strip is connected to the base unit. If a sensor strip is detected, the MCU program initializes each of the IMU sensors with a sensor configuration received by the computer program. This configuration includes, among others, the measurement frequencies and the full scale ranges for accelerometer and gyroscope. The ACC and GYR measurements can be as fast as 1 kHz, while the MAG is limited to 100 Hz. Collecting the raw IMU data reliably from the base unit and up to 5 sensor strips, which may run at different frequencies, is challenging, which is why the internal 512 byte buffer of the MPU9250 is used.
Each MPU9250 uses its internal 1 kHz clock to generate the desired measurement frequencies for the ACC and GYR. The MAG is independent from ACC and GYR and operates with fixed 100 Hz. The measurements for ACC, GYR, and MAG are taken automatically by the MPU9250 controller and added to the first in, first out (FIFO) buffer. This ensures a stable measurement frequency, regardless of delays or jitter within the MCU program. This is important if the raw IMU data is later integrated over time.
The sensor strips are attached to the finger segments with double-sided medical tape, as depicted in Figure 3. Other ways of attachment, like rings, or clamps, were also investigated but were found to be not as good as the medical tape. However, silicon caps for the fingertip are also feasible, if the subject’s sense of touch is not important for the application.
2.3 Joint angle estimation
The orientation of the IMUs is estimated by sensor fusion of the ACC, GYR, and MAG data. Since the system is likely to be used in indoor environments and near ferromagnetic materials, we employ an algorithm that reduces the influence of non-homogeneous magnetic fields by assuring that the inclination (roll and pitch) portion of the orientation remains unaffected .
The attachment of the individual sensors to the body is predetermined by the sensor strip layout and by the layout of the base unit. The coordinate systems of ACC, GYR, and MAG are therefore transformed to follow the ISB recommendations for the hand and forearm . The estimated IMU orientations coincide with the finger segment orientations as defined by the ISB and are represented as quaternions Segiq. They describe the segment orientation relative to a common reference frame defined by gravity and magnetic north.
To obtain the joint angle αA between Segment 1 and 2, the relative quaternion is calculated from the two segment orientations surrounding the joint A by
where is the orientation of the proximal segment and is the orientation of the distal segment. This relative quaternion can be decomposed into Euler angles given a sequence of rotation axes. For the hand and wrist, intrinsic z-x’-y” Euler angles are used to obtain the joint angles specified by the ISB.
Writing the quaternion as
yields the following z-x’-y” Euler angles:
α: flexion (pos.), extension (neg.),
β: adduction (pos.), abduction (neg.),
γ: pronation (pos.), supination (neg.).
For the approximate 1 D joints (DIP, PIP, and TIP), the angles β and γ are negligible, and γ is close to zero for the approximate 2 D joints (MCP).
The accuracy of the estimated joint angles is influenced by three major factors: the calibration of ACC, GYR, AND MAG; the accuracy of the orientation estimation algorithm; the precision with which the sensors are attached to the finger segments. While algorithms exist that allow to reduce the latter , no such methods are employed at the current state.
3 Preliminary results
The following experimental data was obtained with Simulink (MathWorks, Natick, USA) and post-processed with Matlab. The developed Simulink block for the hand sensor system allows for a detailed configuration of the modular hardware and the MPU9250 setup. The measurement frequency for the following preliminary results was 100 Hz. The full scale range was 4 g for the ACC and 1000 deg/sec for the GYR.
Figure 4 shows the joint angles for single segment flexion, using a mechanical finger model that allows moving the joints one by one.
Note that the angles are close to zero for a straight finger, i.e. the sensor axes align well with the anatomical axes. Note furthermore that, although rotating the middle or proximal segment implies moving the distal segment within the global frame, the DIP joint angle remains close to 90 ° for t ∈ [50, 140].
Figure 5 shows the original and the reconstructed finger posture of a pinch grip. The reconstruction uses only the measured joint angles. The segment lengths were taken from an average hand model. The reconstructed pose matches the original pose well, despite a small gap between the fingers.
4 Discussion and conclusion
The presented hand sensor system enables the assessment of hand motor function and feedback applications, e.g. in FES rehabilitation of stroke survivors. The modular sensor strip design reduces the system complexity and setup time for common rehabilitation tasks, where tracking of all fingers or finger segments is not necessary. The achieved accuracy is found to be sufficient for current FES applications but will be improved in a future refinement of the sensor calibration and algorithms. Exploiting the kinematic constraints of the joints might help to eliminate angular offsets caused by potential sensor misalignment or environmental disturbances .
Furthermore, we plan to reduce the size of the base unit, optimize the current layout, and improve the enclosure to meet the clinical requirements.
An in-depth evaluation using an optical reference system is subject of current research, as well as a user survey.
Research funding: This work was funded within the research project BeMobil, which is supported by the German Federal Ministry of Education and Research (FKZ 16SV7069K). Conflict of interest: The Authors state no conflict of interest. Informed consent has been obtained from all individuals included in this study. Ethical approval: The research related to human use complies with all the relevant national regulations, institutional policies and was performed in accordance with the tenets of the Helsinki Declaration, and has been approved by the ethics committee of the Berlin Chamber of Physicians (number Eth-25/15).
Cobos S, Ferre M, Aracil R, Ortego J, Angel M. Simplified Human Hand Models for Manipulation Tasks. In Cutting Edge Robotics 2010. InTech Sept; 2010. Google Scholar
Dipietro L, Sabatini AM, Dario P. A survey of glove-based systems and their applications. IEEE Trans Sys Man Cybernet Part C: Applic Rev. 2008;38:461–82. Google Scholar
Kortier HG, Sluiter VI, Roetenberg D, Veltink PH. Assessment of hand kinematics using inertial and magnetic sensors. J Neuroeng Rehabil. 2014;11:70. Google Scholar
Salchow C, Valtin M, Seel T, Schauer T. Development of a feedback-controlled hand neuroprosthesis: FES-supported mirror training. In Automed Workshop, Wismar Sep; 2016. Google Scholar
Wu G, van der Helm FC, Veeger HE, Makhsous M, Van Roy P, Anglin C, et al. ISB recommendation on definitions of joint coordinate systems of various joints for the reporting of human joint motion–Part II: shoulder, elbow, wrist and hand. J Biomech. 2005;38:981–92. Google Scholar
Seel T, Ruppin S. Eliminating the effect of magnetic disturbances on the inclination estimates of inertial sensors. Proc. of IFAC World Congress 2017 (submitted); 2017. Google Scholar
Seel T, Schauer T, Raisch J. Joint axis and position estimation from inertial measurement data by exploiting kinematic constraints. Proc. IEEE Multi-Conference on Systems and Control; 2012. p. 45–9. Google Scholar
About the article
Published Online: 2017-03-08
Published in Print: 2017-03-01
Citation Information: Current Directions in Biomedical Engineering, ISSN (Online) 2364-5504, DOI: https://doi.org/10.1515/cdbme-2017-0005.
©2017 Markus Valtin et al., licensee De Gruyter.. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. BY-NC-ND 4.0