Nano-modification is considered to be a prospective method to improve material performance. In this paper, oil paper composite insulating materials are prepared respectively by impregnating paper in nano-modified insulating oil (NMIO) and impregnating nano-modified-paper in pure oil, that is nano-oil impregnated paper (NOIP) and oil impregnated nano-modified-paper (OINMP). Then, we determined the breakdown characteristics of NMIO, NOIP, and OINMP. By analyzing the pore size characteristics, the mechanism of nano-particles improving the insulation properties is explored. The results indicate that the 50% breakdown voltage of OINMP reaches its crest (up to 50% compared with unmodified paper) when the doping concentration of Al2O3 nano-particles is 2%. However, Al2O3 nano-particles show limited effect on the 50% breakdown voltage of NOIP. Compared with fresh paper, Al2O3 nano-particles fill up the internal space of some large pores, which reduce the number of large pores (5000 nm~10000 nm) in nano-modified-paper (NMP), thus inhibiting the collision ionization in the oil pores. The above effects are considered to be the main reason to improve the electrical performance of oil paper composite insulation.