Track irregularity is one of the primary factors contributing to excessive vibration and noise in rail transit systems, necessitating cost-effective and efficient identification methods. Existing approaches for identifying track irregularities using in-service train dynamic responses primarily fall into two categories: model-driven and data-driven methods. While model-driven techniques require precise inverse dynamic modeling and accurate vehicle parameters, data-driven methods depend on extensive high-quality datasets and often suffer from interpretability limitations. These constraints impede the practical implementation of track irregularity identification. To overcome these challenges, this study introduces a novel Physics-Informed Neural Network (PINN) framework, termed the Vehicle Inverse Dynamics Neural Network (VIDNN). Unlike conventional PINNs that merely embed physical constraints into the loss function, VIDNN structurally integrates the Multilayer Perceptron (MLP) with the vehicle inverse dynamics model (IDM). Specifically, we first develop a vehicle IDM for track irregularity estimation using vehicle body accelerations, grounded in multibody dynamics theory. The VIDNN architecture is then systematically designed based on this IDM. Numerical simulations and field experiments demonstrate the effectiveness of VIDNN in accurately identifying track irregularities. By rigorously adhering to vehicle inverse dynamics principles in its forward propagation, VIDNN inherently ensures computational interpretability.