China is at a critical stage of energy transition, and improving total factor energy efficiency (TFEE) is an important way to achieve low-carbon development. However, China’s TFEE is still lower than that of developed countries, and there are significant inter-provincial and industrial differences. Considering the dual heterogeneity of regions and industries, this paper uses a modified three-level meta-frontier slacks-based model to analyze the TFEE and the energy-saving potential of the three industries in China’s provinces from 2001 to 2020. The modified method makes the TGR results free from anomalies without changing the original definition of technology gap rate (TGR) and the order of calculation. In addition, this method does not present infeasible solutions. The empirical results show that: First, China’s TFEE has remained low (<0.5) relatively stable overall and shows a fluctuating downward trend until 2018. Second, the eastern region and the primary industry have a higher regional and industrial TGR, as well as TFEE. Finally, although eastern region’s TFEE is greater than that of the other regions, the huge amount of energy input makes its energy-saving potential greater than any other region. Each region and industry should develop appropriate measures to improve TFEE according to their own situation.