The dynamics problem of non-smooth rigid body system considering friction and collision is always a hot and difficult problem in the field of multi-body system dynamics. Different from the general Lagrangian method for constrained systems, the Udwadia-Kalaba method (U-K) decouples the generalized acceleration and the generalized constraint force of the system. The generalized constraint force is obtained directly by solving the explicit equation, and then the generalized acceleration is obtained. It provides a new research idea and method for the dynamics and control of constrained systems. The purpose of this paper is to explore the extension and application of U-K method in the dynamics and control of non-smooth unilateral constraint rigid body systems. Because of the coupling relationship between normal force and tangential force in the expression of Coulomb friction force, when considering the dynamics problem of rigid body system with friction, the non-ideal force expression in U-K equation can not reflect the effect of ideal constraint force. Firstly, the U-K equation is modified. Then, based on the modified U-K equation, the dynamic model and numerical solution scheme of unilateral constraint system with sliding-viscous and collision states change are established. Finally, an example shows the feasibility of the U-K method in the non-smooth unilateral constraint rigid body system.