Monitoring ground surface large displacements in the three-dimensional (3D) space is important for scientifically understanding and controlling deformation-related geohazards. It has become a promising way to estimate surface 3D displacements from airborne or spaceborne multi-temporal point cloud datasets using point cloud alignment techniques such as the typical point-to-plane iterative closest point (ICP) algorithm. However, the typical ICP algorithm aligns point clouds based on geometric features only, causing a poor robustness in the absence of sufficient topographic structures (e.g., for flatten terrain). In this study we proposed a new algorithm so-called Weighting Hue-based ICP (WHICP) for estimating surface 3D displacements from multi-temporal colored point clouds generated from unmanned aerial vehicle (UAV) stereo photogrammetry. Firstly, a variant of the ICP algorithm named hue-based ICP is proposed where geometric and temporally stable color features are both used to point cloud alignment. Then, a multi-window weighted framework was presented to further process the hue-based ICP-aligned point cloud to generate robust estimates of surface 3D displacements. The WHICP algorithm was tested in a coal mine in Tangshan city, China, where large displacements occurred in flatten terrain. The results show that the mean accuracy of the WHICP-estimated 3D displacements is about 0.03 m, indicating an improvement by 84% than that of the typical ICP-estimated 3D displacements.