SynthNet: Leveraging Synthetic Data for 3D Trajectory Estimation from Monocular Video

Morten Holck Ertner, Sofus Konglevoll, Magnus Ibh, Stella Graßhof

Research output: Conference Article in Proceeding or Book/Report chapterArticle in proceedingsResearch

Abstract

Reconstructing 3D trajectories from video is often cumbersome and expensive, relying on complex or multi-camera setups. This paper proposes SynthNet, an end-to-end pipeline for monocular reconstruction of 3D tennis ball trajectories. The pipeline consists of two parts: Hit and bounce detection and 3D trajectory reconstruction. The hit and bounce detection is performed by a GRU-based model, which segments the videos into individual shots. Next, a fully connected neural network reconstructs the 3D trajectory through a novel physics-based training approach relying on purely synthetic training data. Instability in the training loop caused by relying on Euler-time integration and camera projections is circumvented by our synthetic approach, which directly calculates loss from estimated initial conditions, improving stability and performance.\\ In experiments, SynthNet is compared to an existing reconstruction baseline on a number of conventional and customized metrics defined to validate our synthetic approach. SynthNet outperforms the baseline based on our own proposed metrics and in a qualitative inspection of the reconstructed 3D trajectories.
Original languageEnglish
Title of host publicationProceedings of the 7th ACM International Workshop on Multimedia Content Analysis in Sports
PublisherAssociation for Computing Machinery
Publication dateOct 2024
ISBN (Electronic)9798400711985
Publication statusPublished - Oct 2024

Fingerprint

Dive into the research topics of 'SynthNet: Leveraging Synthetic Data for 3D Trajectory Estimation from Monocular Video'. Together they form a unique fingerprint.

Cite this