Health and Behavior: Wearables and Data Analytics for Personalized Therapeutic Management
REU Scholar: Roger Tawfik
REU Scholar Home Institution: University of Massachusetts - Amherst
REU Mentor: Dr. Behnaz Ghoraani
From Lab Desktop to Phone: Automated Quality Control for Gait Video Using 3D Pose Estimation and Large Language Models
View slide presentation for From Lab Desktop to Phone: Automated Quality Control for Gait Video Using 3D Pose Estimation and Large Language ModelsWatch video presentation for From Lab Desktop to Phone: Automated Quality Control for Gait Video Using 3D Pose Estimation and Large Language Models
Video-based gait monitoring could extend far beyond the clinic if a recording captured somewhere like a satellite site, or just a phone in a participant's hand, could be trusted the way a supervised lab session is. But unsupervised recordings fail predictably: poor framing, bad lighting, blur, occlusion, extra people in frame, and a corrupted recording silently corrupts every gait metric derived from it. Clinical gait pipelines have no automated way to reject an unusable video before it reaches analysis, so I built that missing front end: a quality-control (QC) pipeline that ingests a gait recording, runs eight gating checks over lightweight 3D human pose estimation, and uses a large language model (LLM) to tell the participant in plain language whether the recording is usable and how to re-record it. Failing videos loop back for re-recording while passing videos move on to gait-metric computation via MeTRAbs 3D pose estimation. I first built and validated the pipeline on a desktop using a watched-folder file-watcher. I then ported it to run natively on Android and iOS, achieving byte-identical verdicts against a frozen desktop reference set. This ensures the same clinical decision holds whether the recording came from lab hardware or a phone in a clinician's hand. With the system built and running cross-platform, the benchmarking phase is underway, testing the pipeline's checks and thresholds against real ground truth instead of just trusting them by construction. That means auditing each check against real recorded footage, calibrating thresholds like blur and light against instrumented-walkway (Zenomat) ground truth, and running an IRB-approved human-subject validation study to see how well the pipeline's verdicts agree with a human rater's.
REU Scholar: Jeffrey Huang
REU Scholar Home Institution: Cornell University
REU Mentor: Dr. Behnaz Ghoraani
From 3D Skeletons to Walking Events: Detecting Heel Strikes and Toe-Offs from Ordinary Video
View slide presentation for From 3D Skeletons to Walking Events: Detecting Heel Strikes and Toe-Offs from Ordinary VideoWatch video presentation for From 3D Skeletons to Walking Events: Detecting Heel Strikes and Toe-Offs from Ordinary Video
Accurate identification of gait events, particularly heel strikes and toe-offs, is important for analyzing how a person walks. These events are traditionally measured using specialized equipment such as pressure-sensitive walkways or wearable sensors, but video-based pose estimation may provide a more accessible alternative. This project investigated whether gait events can be detected from time series of estimated 3D joint positions produced from ordinary video. Several interpretable kinematic algorithms and a pretrained deep-learning method were implemented and evaluated against events recorded by a reference system. Performance was assessed using timing error, missed-event rate, and false-event rate. The results showed that relatively simple biomechanical methods could provide strong practical performance, while the more complex deep-learning approach did not automatically perform better on the available data. Methods based on extrema of z-axis velocity and the deep learning method performed best on MAE, the deep learning method had fewest false positives, and the shank angular velocity method had the fewest misses. The comparison also indicated that heel-strike and toe-off detection may benefit from different movement signals. Important challenges included noisy foot tracking, changes in walking direction, turning, occlusion, and ambiguity in published algorithm descriptions. Overall, the findings suggest that kinematic methods are a promising approach for converting video-derived 3D skeletons into useful gait-event measurements, while further work is needed to improve tracking quality, expand the dataset, combine complementary methods, and estimate prediction confidence.