Skeleton-Based Modeling in Badminton - Unlocking Insights for Stroke Recognition and Forecasting

This thesis uses skeleton poses as the primary modality for analyzing badminton video sequences. The topic is approached through three central computer vision tasks: Action recognition, action forecasting, and 3D shuttle reconstruction.
One part of the research explores and develops models that best capture informative motion representations from skeleton sequences to infer the performed stroke. Additional features such as shuttlecock position and player court location are integrated to improve recognition performance.
Another focus is on identifying limitations in stroke recognition, such as limited and inconsistent annotation conventions and type imbalance, and exploring potential solutions. Pretraining, especially self-supervised learning, is explored. The models are pretrained using masked autoencoders, reconstructing parts of the skeleton sequences hidden in the input. The findings show that these approaches improve model performance on downstream tasks like stroke recognition.
Achieving the best model performance is not the only goal of the thesis. Part of the research tries to identify which elements of a player’s stroke motion carry the most descriptive information on a particular action. Through model inspection, the different phases of the stroke motion and various modalities are examined using ablation and qualitative attention studies to determine which offers the most relevant informa tion to the model. The results are compared to the human perspective of analysts and coaches to gauge how the findings could benefit coaches and players in match preparation.
Expanding on this, the thesis investigates the model forecasting capabilities for predicting the next strokes. Usually, in sequence modeling, the next stroke in a rally would be predicted based on the sequence of stroke exchanges up to that point. Here, instead, a model architecture is proposed that learns a stroke representation from the player’s skeleton motion, identity, and shuttle position to condition the prediction probability of the next stroke. The model design reflects the turn-based nature of badminton to capture a basic understanding of the game to make informed predictions.
Using 3D information can extract valuable physical and shot statistics while eliminating ambiguities found in 2D image representations. However, the limited availability of 3D badminton data inhibits the development of effective reconstruction models. A physics-based model trained on synthetic 3D shuttlecock trajectories is proposed to overcome this challenge. The developed model, TrajTrans, predicts the initial 3D conditions based on 2D image projections. The results generalize well to real data by implementing shot filtering criteria of the synthetic data that ensure realistic trajectories.
Ultimately, the findings contribute to the field of sports analytics by providing foundational knowledge and guidelines that can advance the development of future tools for analysts and coaches.