Planetary gearboxes with compact structure, large transmission ratio and accurate transmission are widely used in various large machinery and equipment, but they are prone to local failures due to their long-term operation in harsh environments such as high temperature, high speed and heavy load. Therefore, it is important to carry out the dynamics modelling method under gear fault excitation. This paper addresses the problems in the study of the dynamic characteristics of gearboxes containing broken gears of planetary gears and conducts a study in the coupled fault dynamics modelling. The gear box body as well as the ring gear are flexibly modelled, and a coupled gear-bearing-box dynamics model of the planetary gear box is established on the basis of obtaining modal information. The effect of speed fluctuation caused by gear failure on the vibration response of the gearbox is considered in the simulation, and the dynamic response of the box under planetary gear failure is obtained, and the characteristic frequencies under planetary gear failure are analyzed by frequency spectrum. Finally, the dynamic response test of the planetary gearbox under planetary gear fault is carried out, and the test results are in good agreement with the simulation results, thus verifying the correctness of the established rigid-flexible coupling model and providing a theoretical basis for the fault diagnosis of the planetary gearbox.