From 3D Skeletons to Walking Events: Detecting Heel Strikes and Toe-Offs from Ordinary Video

By Jeffrey Huang
Slide 1: Title slide for From 3D Skeletons to Walking Events: Detecting heel strikes and toe-offs from ordinary video, presented by Suhao Jeffrey Huang

Slide-1

From 3D Skeletons to Walking Events

Detecting heel strikes and toe-offs from ordinary video

Suhao Jeffrey Huang

Mentors: John Forde, Prof. Ghoraani

Slide 2: Title slide featuring the Florida Atlantic University I-SENSE institute logo.

Slide-2

Florida Atlantic University

Florida Atlantic University owl logo. Text reads: FLORIDA ATLANTIC UNIVERSITY. I-SENSE: The Institute for Smarter Cities, Spaces, and Health.

Slide 3: The overall picture, describing why gait timing measurement matters and how video-based systems could be a more accessible alternative to specialized equipment.

Slide-3

The overall picture

  • The timing of each step provides useful information about how a person walks.
  • Traditionally, these measurements may require specialized equipment such as pressure-sensitive walkways or wearable sensors.
  • Video-based systems could offer a more accessible alternative.
Slide 4: Two photographs of traditional gait-measurement equipment: a pressure-sensitive walkway mat in a hallway, and a person wearing multiple wireless motion sensor straps on the arms and torso.

Slide-4

This slide contains no title text, only two photographs illustrating traditional gait-measurement equipment.

Left photograph: a long, rectangular pressure-sensitive walkway mat (a Footscan-brand mat) laid on a tile hallway floor in front of a doorway, used to measure footstep pressure and timing as a person walks across it.

Right photograph: a person's torso and arms, shown wearing several rectangular wireless motion sensor units strapped to the upper arms, forearms, and waist with elastic bands, representing wearable inertial sensors used to track body movement.

Slide 5: What the lab already developed, describing a video-to-3D-skeleton pose estimation pipeline, illustrated with a diagram of a human silhouette with colored skeletal joint lines.

Slide-5

What the lab already developed

  • Video → 3D skeleton over time
    • Pose estimation
  • This gives us a moving 3D skeleton.

Illustration: a black silhouette of a standing person overlaid with a colored skeletal diagram, showing white dots at the head, shoulders, elbows, hands, hips, knees, and feet, connected by colored lines (red, green, yellow, blue, purple) representing the joints and limb segments tracked by the pose estimation system.

Slide 6: The missing step: finding step events, defining heel strike and toe-off and posing the question of whether these events can be detected accurately from the 3D skeleton.

Slide-6

The missing step: finding step events

  • Heel strike (initial contact): the foot makes contact with the ground.
  • Toe-off (final contact): the foot leaves the ground.
  • Can heel strikes and toe-offs be detected accurately from the 3D skeleton?
Slide 7: Two possible approaches for detecting step events: kinematic methods using interpretable rules, and machine learning using a pretrained model.

Slide-7

Two possible approaches

  • Kinematic methods
    • Use interpretable rules based on how the feet and legs move.
  • Machine learning
    • Use a pretrained model to recognize events from examples.
Slide 8: How the methods were evaluated, describing the process of comparing predicted events to a reference system and measuring timing error, missed events, and false positives.

Slide-8

How the methods were evaluated

  • Run each method on the 3D skeleton data.
  • Compare its predicted events with events measured by the reference system.
  • Measure:
    • Timing error: How early or late was the prediction?
    • Missed events: How often was a real step not detected?
    • False positives: How often was an extra event predicted?
Slide 9: Phase shift issue in Zeni, a scatter timeline chart showing predicted events (X) drifting out of alignment with ground-truth events (O) within highlighted windows.

Slide-9

Phase shift issue in Zeni

Chart description: a scatter/timeline plot with several horizontal rows of markers. Each row contains a sequence of circle markers (O, in red or blue, representing ground-truth events) and cross markers (X, representing predicted events from the Zeni method). Two rectangular regions in the plot are shaded yellow to highlight windows where the pattern of O and X markers becomes progressively offset from one another moving left to right, illustrating that the Zeni method's predicted events gradually drift out of phase with the true events over time within each highlighted window.

Slide 10: Hybrid approach only requires one annotation per window, a scatter timeline chart with connecting lines and circled anchor points showing how a single manual correction re-aligns predicted events to ground truth for an entire window.

Slide-10

Hybrid approach only requires one annotation per window

Chart description: the same scatter/timeline plot of O (ground-truth) and X (predicted) markers as the previous slide, now with thin vertical connecting lines drawn between paired markers within each shaded yellow window, and a red hand-drawn circle highlighting a single anchor pair of markers near the start of each window. This illustrates that by manually annotating just one corresponding O and X pair per window, the rest of the predicted events in that window can be re-aligned to the ground truth, correcting the phase shift with minimal manual effort.

Slide 11: Zeni and Jasiewicz are the best methods, listing mean absolute error, false positive rate, and miss rate for the top-performing detection methods.

Slide-11

Zeni and Jasiewicz are the best methods

  • Low MAE
    • Jasiewicz (29.8ms)
    • Zeni (30.3ms)
    • Zahradka SK (32.3ms)
  • Low false positive rate (per ground truth event)
    • Zeni (0.17)
    • Jasiewicz (0.19)
  • Low miss rate (per ground truth event)
    • Zeni (0.08)
    • Jasiewicz (0.13)
Slide 12: Hybrid approach decreases MAE of Zeni, a bar chart comparing mean absolute error across fully automatic, hybrid, and phase-shift-removed conditions.

Slide-12

Hybrid approach decreases MAE of Zeni

Bar chart titled "MAE" with a vertical axis ranging from 0 to 70 and three categories on the horizontal axis, each with a single blue bar labeled "MAE" in the legend. "Fully automatic" is approximately 57. "Hybrid approach" is approximately 30. "Phase shift removed" is approximately 23. The chart shows that the mean absolute error of the Zeni method decreases substantially when moving from a fully automatic approach to the hybrid approach, and decreases further when the phase shift is removed.

Slide 13: Hybrid approach decreases miss rate of Zeni, a bar chart comparing missed-event rate across fully automatic, hybrid, and phase-shift-removed conditions.

Slide-13

Hybrid approach decreases miss rate of Zeni

Bar chart titled "Miss rate" with a vertical axis ranging from 0 to 0.8 and three categories on the horizontal axis, each with a single blue bar labeled "Miss rate" in the legend. "Fully automatic" is approximately 0.71. "Hybrid approach" is approximately 0.08. "Phase shift removed" is approximately 0.07. The chart shows that the miss rate of the Zeni method drops sharply once the hybrid approach is applied, with only a small further improvement once the phase shift is removed.

Slide 14: Hybrid approach decreases false positive rate of Zeni, a bar chart comparing false positive rate across fully automatic, hybrid, and phase-shift-removed conditions.

Slide-14

Hybrid approach decreases false positive rate of Zeni

Bar chart titled "False positive rate" with a vertical axis ranging from 0 to 0.9 and three categories on the horizontal axis, each with a single blue bar labeled "False positive rate" in the legend. "Fully automatic" is approximately 0.80. "Hybrid approach" is approximately 0.17. "Phase shift removed" is approximately 0.14. The chart shows that the false positive rate of the Zeni method drops sharply once the hybrid approach is applied, with a small further decrease once the phase shift is removed.

Slide 15: Zeni performs better than machine learning, a bar chart comparing miss rate between the Zeni method and a machine learning method.

Slide-15

Zeni performs better than machine learning

Bar chart titled "Miss rate" with a vertical axis ranging from 0 to 0.7 and two categories on the horizontal axis, each with a single blue bar labeled "Miss rate" in the legend. "Zeni" is approximately 0.08. "Machine learning" is approximately 0.58. The chart shows that the Zeni method has a substantially lower miss rate than the machine learning method.

Slide 16: Zeni performs better than machine learning, a bar chart comparing false positive rate between the Zeni method and a machine learning method.

Slide-16

Zeni performs better than machine learning

Bar chart titled "False positive rate" with a vertical axis ranging from 0 to 0.4 and two categories on the horizontal axis, each with a single blue bar labeled "False positive rate" in the legend. "Zeni" is approximately 0.17. "Machine learning" is approximately 0.35. The chart shows that the Zeni method has a lower false positive rate than the machine learning method.

Slide 17: Limitations, noting the small dataset size and ambiguity in some published algorithm descriptions.

Slide-17

Limitations

  • The dataset was small (~20 subjects)
  • Some published algorithm descriptions leave implementation details ambiguous.
Slide 18: Next steps, outlining plans to test on more participants, combine methods, train models on this dataset, and create confidence scores.

Slide-18

Next steps

  • Test on more participants and walking conditions.
  • Combine methods.
  • Train machine learning models specifically on this dataset.
  • Create confidence scores to identify uncertain predictions.
Slide 19: Acknowledgments, crediting the NSF REU Site in Sensing and Smart Systems for funding this work.

Slide-19

Acknowledgments

This work was supported through the NSF REU Site in Sensing and Smart Systems, funded through NSF Award CNS-2447437.

Last slide: Contains plain text stating 'End of presentation. Click the right arrow to return to beginning of slide show.'

End of Presentation

Click the right arrow to return to the beginning of the slide show.

For a downloadable version of this presentation, email: I-SENSE@FAU.

Additional Information
The Institute for Smarter Cities, Spaces, and Health was established in early 2015 to coordinate university-wide activities in the Sensing and Smart Systems pillar of FAU’s Strategic Plan for the Race to Excellence.
Address
Florida Atlantic University
777 Glades Road
Boca Raton, FL 33431
i-sense@fau.edu