Motion tracking glove for augmented reality and virtual reality

Abstract This project designs a smart glove, which can be used for motion tracking in real time to a 3D virtual robotic arm in a PC. The glove is low cost with the price of less than 100 € and uses only internal measurement unit for students to develop their projects on augmented and virtual reality applications. Movement data from the glove is transferred to the PC via UART DMA. The data is set as the motion reference path for the 3D virtual robotic arm to follow. APID feedback controller controls the 3D virtual robot to track exactly the haptic glove movement with zero error in real time. This glove can be used also for remote control, tele-robotics and tele-operation systems.


Introduction
In recent years, different augmented and virtual reality tools have been developed. It is due to the new virtual reality wave from 2012, when Oculus developed a new virtual reality headset with "full immersion effect" of 640 by 800 pixels for each eye of the user. This event triggered the beginning of the new virtual reality technological development and the rapid growth of investments. From 2015, virtual reality technologies have become the new technological revolution and many well-known companies are being engaged.
Virtual reality has become an industry in which technologies evolve at the same time with the applications. For example, if there is a virtual reality helmet or glasses, there must be applications for them to interact and to operate. That is why the applications for virtual reality are rapidly covering a wide variety of fields: movies, games, online streams, social networks, medicine, education, trading, manufacturing, and engineering.
Virtual reality glasses immerse their users into the virtual reality, but different types of interaction with virtual reality, such as gamepads and joysticks, interfere with the effect of presence. Haptic gloves have therefore become an excellent solution. However, one of the disadvantages of haptic gloves is their high price, which sometimes exceeds even the cost of the original glasses for virtual reality. Further, haptic gloves can offer the user to feel the objects in the virtual environment using haptic feedback, which gives an in-depth effect of presence. Haptic gloves can have various applications in remote control and humanrobot interaction. It is also possible to interact not only with virtual reality but also with augmented reality as well. For example, this glove can be used together with virtual reality headset and stereo camera for remotely controlling a robot arm.
A survey on environments and system types of virtual reality technology in science, technology, engineering, and mathematics is presented in [1]. It introduces detailed information on virtual reality systems and makes comparison among them. Another review on immersive environments and virtual reality is given in [2], which provides the latest advances in communication and simulation in the virtual reality world. A review on general concept and application of virtual reality haptic technology is briefed in [3]. This paper summarizes latest experimental publications, available studies and development of research projects in haptic virtual reality system.
Related to application of the haptic glove on remote control, a tele-operation system, [4] presents a novel scheme for a tele-operation system in variant time delays and environment uncertainties. The system can be used by a medical doctor examining remote patients. Reference [5] presents state of the art in haptic devices to improve in-teractions of tools applied in the evaluation of product design.
A project in [6] introduces a passive deformable haptic glove to support 3D interaction in mobile augmented reality environments. The glove is designed with a digital foam sensor placed under the palm to support precise direct touch manipulation. Reference [7] presents haptic links as electro-mechanically actuated physical connections that can improve the haptic rendering of two-handed objects and interactions in virtual reality.
Other latest development on wearable haptic gloves in augmented reality is reviewed as follows. A study on evaluation of wearable haptic systems for the fingers in augmented reality applications is presented in [8]. A novel shape-changing input device providing passive haptic feedback is introduced in [9]. A mobile, wearable haptic device for simulating the grasping of rigid objects in virtual reality interface is briefed in [10]. A design and validation of a wearable glove-based multi-fingers-motion capture device are presented in [10]. A study on haptic stimulation glove for fine motor rehabilitation in virtual reality environments is provided in [11]. A paper on wearable haptic interface for simulating weight and grasping in virtual reality is presented in [12].
Most of the above haptic gloves are expensive and based on flex sensors, which change their resistance by bending. These sensors are expensive and cannot register resistance change in several bending points. One low cost sensor glove with vibro-tactile feedback and multiple finger joint and hand motion sensing for human-robot interaction is discussed in [14]. This low cost glove is made with a price tag of 300 . Another low cost sensor glove with force feedback for learning form demonstrations using probabilistic trajectory representations is introduced in [15]. Updated references on control system are referred to in [16,17]. This glove is fabricated with price of 250 .
In our project, a low cost glove for real time interaction with virtual robotic arm is designed with price tag of less 100 , that enables the students to develop their applied research project with virtual and augmented reality software in various applications.
This project is an initial step toward real project on hand gesture recognition in tele-health applications to improve the quality of patient care locally and remotely. Human hand gesture recognition is considered as an important way for patient-doctor tele-interfaces. This leverages cutting-edge sensor-based technologies, internet of things (IoT) and big-data analytics to enable lonely elderly and disable persons to remotely communicate with caregivers and healthcare with their hand gestures. Results of this study allow integrating intelligent medical wearable de-vices with hand gesture recognition in tele-healthcare applications. It improves the healthcare quality services for remote patients and lonely elderly people as it assures them that one hand gesture could be sufficient for informing emergency aids. This reduces the time and costs significantly of the healthcare system.
The following are the contents of this paper. Section 2 presents the design of glove. Section 3 briefs about the data collection and angle calculation. Section 4 introduces the 3D Matlab simulations and experimental results. Finally section 5 draws conclusion and future recommendations.

The glove design
Based on anatomical and medical hand analysis in previous researches and in [11], the hand skeleton model has 23 degrees of freedom (DOFs) as shown in Figure 1. Each of the four fingers has four DOFs. The Distal-Interphalangeal (DIP) and Proximal-Interphalangeal (PIP) joints both have one DOF, and the remaining two DOFs are located at the Meta-Carpophalangeal (MCP) joint. Different from the four fingers, the thumb has five DOFs. Two DOFs are at the Trapeziometacarpal (TM) joint, also referred to as the Carpometacarpal joint (CMC), and two are at the MCP joint. The remaining one DOF of the thumb is at the Inter-phalangeal (IP) joint. The basic flexion/extension (F/E) and abduction/adduction (Ab/Ad) of the thumb and fingers are performed by the articulation of the 21 DOFs. As shown in Figure 2, the F/E motions are used to describe rotations toward and away from the palm, which occur at every joint within the hand. The abduction is the movement of separation (e.g., spreading fingers apart), and the adduction motion is the movement of approximation (e.g., folding fingers together). The Ab/Ad only occurs at each finger's MCP joint and at the thumb's MCP and TM joints. Another two internal DOFs are at the base of the fourth and fifth (ring and little finger) metacarpals, which perform the curve or fold actions of the palm.
The main goal of the project is to develop a haptic glove using only the inertial measurement units (IMUs), which provide realistic hand and finger motions. For this purpose, IMU MPU-6050 is chosen. The MPU-6050 device has built-in accelerometer and gyroscope sensors and raw data from these sensors can be combined to calculate the moving angle of the arm and each finger.
The main microcontroller for this glove is selected with STM32-F413ZH, a high-performance Arm Cortex 32bit core. A total of 16 MPU-6050 devices are installed in this glove as shown in Figure 3. These devices provide raw data of gravity acceleration and angular velocity, which can be converted into moving angles using trigonometric equations. As instructed in [18] for tracking of both fast and slow motions, the parts feature a userprogrammable gyroscope full-scale range of ±250, ±500, ±1000, and ±2000 ∘ /sec (dps) and a user-programmable accelerometer full-scale range of ±2g, ±4g, ±8g, and ±16g. Communication with all registers of the devices is performed with I2C protocol of 100 kHz in normal mode or in fast-mode of 400 kHz.
In our glove, each IMU MPU-6050 device has 2 different addresses and a multiplexer 74HC4067 is used. This multiplexer can be controlled by microcontroller STM32, which can send registers to 4 multiplexer inputs and reads data sequentially from it. Then, the microcontroller gathers data and sends data package to computer over DMA USART communication protocol. The DMA USART can be used in several ways. For example, the microcontroller can be connected directly via TTL-USB converter or distantly via Bluetooth module. After receiving from sensors and processing in microcontroller, data is used to calcu-  late the real angle movement. The control system diagram is shown in Figure 3. Then, a glove prototype is fabricated and shown in Figure 4.

Angle calculation and data collection
In the initial position, when the gravity force acts only in the opposite direction along the z axis, it is seen that all angles are equal to 0. From the properties of the sine and cosine that while the sensitivity along one axis decreases, it increases in the other. The rotation angles around the axes x, y, and z can be detected in the accelerometer sensors and calculated by the following formulas: = arctan where Ax -gravity force vector along x axis, g; Ay -gravity force vector along y axis, g; Az -gravity force vector along z axis, g. The rotating angles can be also detected from the gyroscope sensors. The gyroscope provides the angular velocity. Therefore, it is less sensitive to noises and vibrations. To obtain the angle from the angular velocity, the gyroscope data must be integrated and an initial angle (i.e., zero angle of the gyroscope) added to the output angles. Integration is performed following the algorithm: where α (t) -current angle at time t, deg; α (t − 1) -angle at the previous time period, deg; rawData -raw data from the accelerometer, deg/ms; dt-step time, ms. Unlike in the acceleration sensor, the angle position of the gyroscope is slowly drifting away from the stable position. Figure 5 shows that in a stable position at 0 degrees, the angle in the acceleration sensor is stable, while the angle from the gyroscope is slowly moving away. Then, gyroscope sensors need to be reoriented periodically by a magnetic compass reference offsetting the drifting bias and calibrating it when it stops and re-sample the rate of the gyroscope.
A complementary filter is built using the combination of two set of data, one set from gyroscope and the other from the accelerometer. The following algorithm is used for the complementary filter: where θ accel -angle from the accelerometer, deg; θgyro -angle from the gyroscope, deg; θ ltered -filtered angle, deg; FK -complementary filter coefficient can be adjusted from 0 to 1.
The filter data is tested in a stable position. As a result, in Figure 6, the filtered data looks smoother and less vibrated than the data from the accelerometer. Next, this filtered data is tested with a real dynamic movement.
The filtered data is now tested with the arm moving from +90 degrees to −90 degrees. Figure 7 shows that the filtered data is better than accelerometer data.

Matlab simulations and experimental results
For the visualization of the prototype, it is necessary to design a simulation in an appropriate software. In this study, Matlab Simulink is selected since it is possible to build physical components in Simulink environment and implement receiving data from Simscape add-on blocks. A 3D solid-work arm is designed and transferred into Matlab Simscape. Figure 8 shows how the 3D solid-work arm model works in Matlab with full elements of hand, fingers, elbow, and wrist. Figure 9 shows the 3D solid-work arm model transferred into Matlab Simulink with three blocks: serial ports, joints, and the 3D model. Angular data from the haptic glove is transferred to PC via UART DMA at block serial ports. The data is set as reference paths and fed into block joints and control the virtual arm in block 3D Model to track. Errors of the virtual arm movements and the reference paths are fed back to a PID controller in block joints. This PID controller controls the 3D virtual arm tracking which exactly matches the haptic glove movement in real time with zero error.
The glove prototype with all the sensors, microprocessor and communication system is fabricated. As mentioned earlier that the aim of this project is to design a low cost haptic glove. At this final step, the total expense for the prototype is less than 100 . The cost analysis is shown in Table 1.
The haptic glove can work now with the real time 3D virtual robotic arm as shown in Figure 10. This haptic glove can also be used for remote control, tele-robotics and teleoperation systems.   The following are some experimental results of the rotating angle data recorded from the arm, index finger and middle finger. This data is calculated as reference paths and control the virtual robot tracking exactly on those paths in real time by a PID controller. Figure 11 shows the rotating angle of the arm around the axes x, y, and z. Figure 12 shows the rotating angle of the index finger around the DIP, MCP and PIP joint.    Figure 13 shows the rotating angle of the middle finger around the DIP, MCP and PIP joint.
In addition, a testing video is recorded and shows that the prototype works correctly and corresponds to the hand movement in real time. The YouTube video can be seen at the following link: https://www.youtube.com/watch?v=hRFgJV7qrsU.
In this study, we have used 16 IMU sensors to detect and measure the real movement of human hand and fingers. These movement data is used to run a 3D virtual robotic arm in PC to verify the real time tracking with free error feedback. The purpose of this study was to verify the ability of human gestures detections and recognition using simple IMU sensors and/or EMG in smart communication systems. It will be used as an alternative option for remote patients and lonely elderly people sending their messages or interfaces to their doctors healthcare service providers and emergency medical aid centers.

Conclusion
This haptic glove can provide 23 degrees of freedom and correspond to a real human arm. The haptic glove fits well with the hands of different people. Simulations show that the glove can provide accurate angular movement of the arm, the joints and can correctly control in real time, the movement of the virtual robot in PC. The cost for this haptic glove is less than 100 . Applications of this haptic glove can be applied for simulating the virtual robot or for remotely controlling a real robotic arm. In the future, the prototype can be installed with MEMS sensors to improve the speed and accuracy of the angular detection. In addition, the 3D solid work arm can be re-designed more re-alistically which can better correspond to the real human hand.