Aiming at the problem that dim small targets are submerged to complicated background in infrared images, it is difficult to complete extraction from background and noise clutter. An improved vibe algorithm is proposed for small target detection and tracking. First, target areas are extracted and stored by using vibe algorithm in every frame of video, meanwhile local multi-gradient filter are used to detect and store prominent edge information in each same frame of video. Then, fusion image is obtained through vibe algorithm and multi-gradient filter. Finally, a threshold separation technique is used to further eliminate background clutter and extract small targets. The experimental results show that proposed algorithm can quickly eliminate ghosts and is effective for detecting moving small targets, and compare to other background difference method, gaussian mixture model, and experimental evaluation results show that our method outperforms vibe ,background difference method and gaussian mixture model methods in terms of both tracking accuracy and computation speed for detection infrared small targets.