Aiming at the problems of difficulty in matching homonymous points and difficulty in obtaining horizontal movement information when using point cloud data for surface deformation analysis, a point cloud deformation analysis algorithm with strong adaptability to the terrain and without the need for artificial feature marking is proposed: a point cloud feature area deformation analysis algorithm based on curvature and rectangular segmentation. In this paper, we take the triangular facets in the triangular mesh as the research object for deformation analysis, and put forward the deformation analysis algorithm of point cloud Hausdorff distance triangular mesh. In order to verify the point cloud deformation analysis algorithm proposed in this paper, a three-dimensional similar material simulation experiment was carried out to develop a complete set of observation and data processing technology system of three-dimensional similar material model, so as to better simulate the movement and deformation of the ground surface in the process of mountainous coal mining and to study the ground surface movement law of mountainous coal mining.