The performance of miniature mechatronic systems can be significantly improved by incorporating the magnetorheological (MR) fluid-based rotary brake. However, the small-size effect becomes evident in the process of miniaturization. The mechanical movement of the shaft shifts from a pure rotation to a compound motion of rotation and swing, resulting in a variable MR action gap. The paper proposes a comprehensive dynamic model to explain the interaction mechanism among the compound motion, variable MR action gap and speed reduction performance of a miniature MR brake. A high-speed camera-based dynamic performance testing system is developed to capture the motion state of the shaft and test the speed reduction performance. The compound motion of rotation and swing is observed when the rotational speed is above 3500 r/min. As the rotational speed n increases, the shaft’s (with a diameter of 1.6 mm) swing angle reaches as high as 2.216°. Results show that the behavior of the miniature MR brake can be well explained by the proposed comprehensive model. Compared with the traditional model based on a pure rotation, the accuracy of this proposed model can be increased more than 39.59% by considering the compound motion of rotation and swing. At a desired speed of 14000 r/min, the maximum error (87 r/min) of the proposed model is less than 8.5% that of the pure rotational model (1027 r/min). We believe this work is significant for precisely describing the dynamics of miniature MR brake and promoting its application in miniature mechatronic systems.