As a high-power power source, the output mode of the gyrotron is usually TE0n mode. As a part of high power output system, modal converter plays a very important role. This paper mainly introduces the method of converting TE01 mode to TE11 mode. The traditional TE01-TE11 analog converter is calculated by coupling wave theory and designed by axis function. In this paper, based on the iterative method, genetic algorithm is used to optimize the parameters of the axis function to obtain a more efficient modal converter. The genetic algorithm can optimize the function in the global scope, and the result is more accurate and effective.