The position loop (outer loop) and the attitude loop (inner loop) of a quadrotor are respectively a second-order linear system and a nonlinear system, and the essence of backstepping control is a proportional-derivative (PD) control. When the quadrotor is affected by unmodeled dynamics, parametric uncertainty and aerodynamic disturbance, we design a kind of inner and outer loops hierarchy structure controller to steadily track the quadrotor trajectories, this hierarchy structure controller is that internal model control/backstepping sliding mode controller. Specifically, the backstepping sliding mode controller with strong robustness is used in the inner loop, and two degrees of freedom internal model controller (IMC) only with a feedforward controller adjustment parameter and a feedback controller adjustment parameter is adopted in the outer loop. A stability analysis demonstrates that the close-loop system with this hierarchy structure controller is stable. The effectiveness of the control strategy is tested by tracking a conical spiral and a flight path existing in the literature. Simulation results verify that the quadrotor can steadily track the given fast-varying trajectories and has strong robustness to parameter uncertainties and constant disturbances.