Mobile devices such as smart phones, tablets and laptops are increasingly integral in our daily lives. Traditional interaction paradigms like on-screen keyboards can be an inconvenient and cumbersome way to interact with these devices. This has led to the advancement of bio-signal interfaces such as speech based and hand gesture based interfaces. Compared to speech input interfaces, hand gesture devices can be more intuitive in resembling physical manipulations related to spatial tasks such as navigation on a map or manipulating a picture.

Electromyography (EMG) is well suited for capturing static hand features involving relatively long and stable muscle activations. At the same time, inertial sensing can inherently capture dynamic features related to hand rotation and translation. This project explores a hand gesture recognition wristband based on combined EMG and IMU signals. Preliminary testing was performed on four healthy subjects to evaluate a classification algorithm for identifying four surface pressing gestures at two force levels and eight air gestures. Average classification accuracy across all subjects was 88% for surface gestures and 96% for air gestures. Classification accuracy was significantly improved when both EMG and inertial sensing was used in combination as compared to results based on either single sensing modality.

For details see: Jiang S, Lv B, Guo W, Zhang C, Wang H, Sheng X, Shull PB, “Feasibility of wrist-worn, real-time hand and surface gesture recognition via sEMG and IMU sensing,” IEEE Transactions on Industrial Informatics, 14(8): 3376-3385, 2018.

Gait retraining to reduce the knee adduction moment is an important treatment method for knee osteoarthritis, but current paradigms do not account for differences in human anatomy and do not allow patients to change walking directions or perceive haptic feedback sensations during gait, which have prevented clinical application. This project investigates human movement control, sensing, and haptic feedback, to explore the scientific basis for a wearable system to improve knee osteoarthritis based on novel concepts of data-driven adaptable gait prediction optimization, real-time heading vector estimation, and skin mechanical resonance haptic feedback. The research is conducted based on technical theories and scientific experiments to investigate the following issues: real-time gait biomechanics modeling and wearable sensor fusion, gait prediction optimization modeling and control, and wearable skin stretch feedback device design.

This project focuses on three key problems: 1) inability to accurately predict new gait kinematics to reduce the knee adduction moment, 2) inability to measure human gait kinematics after humans change walking directions, and 3) imperceptible haptic feedback during gait. A wearable sensing and haptic feedback system prototype will be developed to demonstrate key research findings and clinical testing will be performed to establish a scientific basic for practical application. The results of this study are intended to provide a scientific foundation for wearable systems research for gait retraining.

For details see: Xu J, Bao T, Lee UH, Kinnaird C, Carender W, Huang Y, Sienko KH, Shull PB, “Configurable, wearable sensing and vibrotactile feedback system for real-time postural balance and gait training: proof-of-concept,” Journal of NeuroEngineering and Rehabilitation, 14(1): 102, 2017.

Effective stroke rehabilitation typically involves improving both motor function and mental state. Traditional rehabilitation systems are generally expensive, not portable and difficult to operate. As a first step toward overcoming these existing shortcomings, we developed a cellphone augmented reality (AR) rehabilitation system for stroke rehabilitation. Three serious games for upper limb motor function and cognitive training were developed in which patients move their arms to touch virtual targets generated in the three-dimensional AR environment. Patients received visual and vibration feedback after successfully touching each target.

Stroke patients with upper limb dysfunction performed pilot validation testing with the cellphone AR system. Motor function performance was evaluated based on performance in the AR games and mental state was evaluated via questionnaires self-reporting various aspects of mental state. Results showed that motor performance improved by the last trial as compared to the first trial. In addition, the majority of patients reported that they felt ‘relaxed’ and ‘happy’ while playing the AR rehabilitation games. These results demonstrate the potential of cellphone AR rehabilitation to improve upper-limb motor function and mental state for stroke patients and lay a foundation for a future home-based rehabilitation paradigm.

For details see: Song X, Ding L, Zhao J, Jia J, Shull PB, “Cellphone augmented reality game-based rehabilitation for improving motor function and mental state after stroke,” IEEE International Conference on Wearable and Implantable Body Sensor Networks (BSN), 2019.

Hand movement measurements are vital for upper extremity rehabilitation, particularly after stroke. Wearable hand gesture recognition is a promising approach as it has been shown to aid in upper extremity stroke rehabilitation. Wearables can further serve to monitor short- and long-term rehabilitation progress for acute injuries and chronic disease by providing objective patient metrics to enable therapists to optimize such rehabilitation programs. Wearable hand gesture recognition is also fundamental to enabling intelligent upper extremity prosthetics to function in real-life situations.

This project presents a new approach to wearable hand gesture recognition and finger angle estimation based on barometric pressure sensing. A wearable prototype consisting of an array of barometric pressure sensors around the wrist was developed and validated with experimental testing for three different hand gesture sets and finger flexion/extension trials for each of the five fingers. This project demonstrates that a barometric pressure wristband can be used to classify hand gestures and to estimate individual finger joint angles. This approach could serve to improve clinical treatment for upper extremity deficiencies, such as for stroke rehabilitation, by providing objective patient motor control metrics to inform and aid physicians and therapists throughout the rehabilitation process.

For details see: Shull PB, Jiang S, Zhu Y, Zhu X, “Hand gesture recognition and finger angle estimation via wrist-worn modified barometric pressure sensing,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27(4): 724-732, 2019.

Soft and integrated design can enable wearable haptic devices to augment natural human taction. This project proposes a novel, soft, haptic finger-worn wearable device based on compliant and adhesive silicone skin and lightweight twisted and coiled polymer (TCP) actuators using ultra high molecular weight polyethylene (UHMWPE) fibers to provide lateral skin stretch sensations. Recently, silicone elastomers have been used in wearable sensors and in haptic applications for their high compliance or adhesion. TCP actuators have also demonstrated high power to weight ratios, large stroke length, simple mechanism, and inherent softness. Lateral skin stretch is sensitive to small motions and has been used for intuitive proprioceptive feedback applications.

We combined these characteristics to design and manufacture a wearable, functional haptic prototype. Prototype performance was evaluated using an optical tracking system, a force gauge test bench, and compared to vibrotactile haptic feedback in a experiment with healthy participants. Results showed that participant mean reaction times were comparable to those of a vibrotactile feedback system, though task completion times were longer. This project is the first to employ TCP actuators for haptic stimulation and could serve as a foundation for future applications involving soft wearable haptics in gaming, health, and virtual reality.

For details see: Chossat JB, Chen DKY, Park YL, Shull PB, “Soft wearable skin-stretch device for haptic feedback using twisted and coiled polymer actuators,” IEEE Transactions on Haptics, 12(4): 521-532, 2019.

Wearable energy harvesting during gait is an important potential avenue for powering the vast array of recently available wearable devices for clinical and recreational purposes, however current approaches either produce too little power to be useful or increase the human metabolic cost of gait thus preventing practical application. This project will investigate movement control, adaptation, and energy consumption to explore the scientific basis for a regenerative, smart shoe system for harvesting energy during gait based on the novel concepts of comprehensive human mechanical and metabolic energy estimation, gait change prediction, and horizontal foot sole slider energy harvesting. The research will be conducted based on technical theories and scientific experiments to investigate the following issues: human mechanical and metabolic energy gait model, gait change prediction model to convert negative work to positive work, and smart shoe regenerative harvester design. Three key problems are expected to be solved: unknown gait braking energy loss, converting negative work braking energy into positive work propulsion energy, and harvesting gait energy without increasing metabolic cost. A regenerative, smart shoe system prototype will be developed to demonstrate key research findings and gait testing will be performed to establish a scientific basic for practical application. The results of this study will provide a scientific foundation for energy harvesting research for powering wearable devices.

For details see: Xia H, Chen DKY, Zhu XY, Shull PB, “Controlled slip energy harvesting while walking,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(2): 437-443, 2020.

The aim of this project is to develop novel multi-sensor fusion models, which combines wearable sensing data (accelerometer, gyroscope, and magnetometer) to compute clinically important human kinematics during dynamic movement. We initially focus on accurate sensing during gait for foot progression angle (relevant to knee loading for knee osteoarthritis) and trunk sway angle (relevant to walking stability).

One primary problem with the current kinematic estimation algorithms is that gravity and magnetic field vectors are measured in the earth’s reference frame, but gait kinematics must be expressed in the human body reference frame. Thus, a static trial is typically performed prior to the walking trial to align the earth reference frame with the body reference frame. However, this only works when the human body reference frame and the earth reference frame do not change with respect to each other during the entire walking trial. Current research involving wearable kinematic sensing is confined to conditions of straight line walking and doesn’t allow humans to change walking direction. However, in real life walking situations, humans are constantly turning and changing walking directions. 

Thus, research will be performed to form a model to compute human gait kinematics in real-time when changing walking directions based on multiple wearable sensors attached to multiple body segments. First, develop sensor fusion algorithms to combine accelerometer, gyroscope, and magnetometer signals to accurately estimate each body segment at the location of the sensors, which includes solving the drift problem of integrating gyroscope angular velocities, the environment magnetic noise problem of magnetometers not always measuring true magnetic north pole, and the unwanted accelerations problems of accelerometers. Second, combine individual body segment orientations to compute anatomical gait kinematics, which involves solving the problem of inaccurate sensor placement on the body. Third, project all gait kinematics into the reference frame of the human to ensure they are still valid after changing walking directions. Accelerometer, gyroscope and magnetometer data will be fused together, integrated, and Gradient decent, Kalman, and custom filters applied to estimate kinematics. Vicon, marker-based motion capture system is used to validate measurements accuracy. 

For details see: Huang Y, Jirattigalachote W, Cutkosky MR, Zhu XY, Shull PB, “Novel foot progression angle algorithm estimation via foot-worn, magneto-inertial sensing,” IEEE Transactions on Biomedical Engineering, 63(11): 2278-2285, 2016.

Existing wearable devices are typically hard, manufactured with techniques for rigid consumer electronics like smart phones, and are thus often not well suited for use on human skin, which is soft and elastic. We focus on developing elastic substrates which closely match human skin properties. Electronics, sensors, actuators, and wires are embedded in elastic substrates as part of the design process. Soft robotic sensing devices contain accelerometers, gyroscopes, and magnetometers, and haptic devices contain small vibration motors for haptic vibration feedback. 

Design and fabrication of elastic substrates for electronic sensing and feedback devices are challenging due to the inherent properties of flexible/elastic materials. Specific challenges include susceptibility to micro cracks during the deposition process and the potential for buckling and wrinkling of rigid thin films on elastic substrates. Additionally, adhesion of deposited metal on elastic substrates can be challenging given the mismatch in thermal coefficients between the two materials. 

Lithography and electroforming are critical to the fabrication process. First a layer of PDMS is coated on a rigid glass base. This is followed by seed layer deposition and the use of lithography to form a resist layer to define the circuit structure. Electroforming is then used to make the gold metal conductive circuit. The resist layer is then removed and the seed layer etched. The sensors, microcontroller, battery, and other necessary electronic components are then soldered to the circuit substrate to complete the device packaging. Finally, the glass base is released leaving only the elastic substrate.

For details see: Jiang S, Li L, Xu H, Xu J, GU GY, Shull PB, “Stretchable e-skin patch for gesture recognition on the back of the hand,” IEEE Transactions on Industrial Electronics, 67(1): 647-657, 2019.

Resonant frequency skin stretch uses cyclic lateral skin stretches matching the skin’s resonant frequency to create highly noticeable stimuli, signifying a new approach for wearable haptic stimulation. In this project, three experiments were performed to explore biomechanical and perceptual aspects of resonant frequency skin stretch. In the first experiment, skin resonant frequencies were quantified at the forearm, shank, and foot. In the second experiment, perceived haptic stimuli were characterized for skin stretch actuations across a spectrum of frequencies. In the third experiment, haptic classification ability was determined as subjects differentiated haptic stimulation cues while sitting, walking, and jogging. Results showed that subjects perceived stimulations at, above, and below the skin’s resonant frequency differently: stimulations lower than the skin resonant frequency felt like distinct impacts, stimulations at the skin resonant frequency felt like cyclic skin stretches, and stimulations higher than the skin resonant frequency felt like standard vibrations. Subjects successfully classified stimulations while sitting, walking, and jogging, and classification accuracy decreased with increasing speed, especially for stimulations at the shank. This work could facilitate more widespread use of wearable skin stretch. Potential applications include gaming, medical simulation, and surgical augmentation, and for training to reduce injury risk or improve sports performance.

For details see: Shull PB, Tan T, Culbertson HM, Zhu X, Okamura A, “Resonant frequency skin stretch for wearable haptics,” IEEE Transactions on Haptics, 12(3): 247-256, 2019.

This work focuses on the development and validation of a smart shoe for estimating foot progression angle during walking. The smart shoe is composed of an electronic module with inertial and magnetometer sensing inserted into the sole of a standard walking shoe. The smart shoe charges wirelessly and up to 160 hours of continuous data (sampled at 100Hz) can be stored locally on the shoe. Foot progression angle estimations from the smart shoe were compared with estimations from a standard motion capture system. In general, foot progression angle estimations from the smart shoe closely followed motion capture estimations for all walking conditions with an overall average estimation error of 0.1±1.9deg and an overall average absolute estimation error of 1.7±1.0deg. There were no significant differences in foot progression angle estimation accuracy among the seven different walking gait patterns. The presented smart shoe could potentially be used for knee osteoarthritis or other clinical applications requiring foot progression angle assessment in daily life or in clinics without specialized motion capture equipment. The smart shoe could also potentially be used for interactive games, assessing athletic performance, or assisting in rehabilitation. 

For details see: Xia H, Xu J, Wang J, Hunt MA, Shull PB, “Validation of a smart shoe for estimating foot progression angle during walking gait,” Journal of Biomechanics, 61: 193-198, 2017.