Robust Real-Time Garment Fitting from 3D Point Clouds with Physics-Guided Uncertainty and Reliability Monitoring

Abstract

Real-time garment fitting from 3D point clouds is a fundamental capability for telepresence, virtual try-on, and augmented reality. Yet, it remains difficult due to high-dimensional non-rigid deformation, frequent self-occlusion, and the need to maintain physically plausible cloth behavior under noisy measurements. Existing studies have improved geometric registration, learning-based reconstruction, or cloth simulation in isolation, but still lack a unified real-time framework that combines dense vertex-level tracking, uncertainty-aware sequential estimation, and physics-guided regularization for robust garment fitting under noisy and incomplete 3D observations. To address this gap, we propose DenseDecoupled Adaptive Kalman Filtering (DD-AKF), an integrative sequential estimation framework that tracks dense garment vertices in real time while explicitly propagating uncertainty. To make high-dimensional filtering tractable, DD-AKF employs a block-diagonal (decoupled) covariance approximation that reduces per-frame complexity to linear time in the number of vertices while retaining per-vertex uncertainty estimates. Physical plausibility is incorporated by introducing differentiable stretch, bending, and collision energies as soft constraints in a Gaussian-approximate (sequential MAP) update, enabling an adaptive trade-off between sensor fidelity and physics under degraded observations. We also compute an online reliability score based on innovation (residual) statistics to monitor tracking quality and trigger robustification in the presence of occlusion or sensor corruption. Experiments on FAUST, CLOTH3D, and real capture sequences show reduced temporal flicker and penetration compared with representative optimization-based, stochastic filtering, and learning-based baselines, while maintaining competitive geometric accuracy at interactive rates (30+FPS). These results support reliability-aware deployment in practical virtual try-on and AR garment-fitting pipelines using commodity 3D sensors.