The dynamic vibration of the gear coupling-rotor system (GCRS) caused by misalignment is an important factor of low frequency vibration and noise radiation of the naval marine. The axial misalignment of gear coupling is inevitable owing to mass eccentricity, and is unconstrained in axial direction at high-speed operation. Therefore, the dynamic model of GCRS is proposed, considering gear-coupling misalignment and contact force in this paper. The whole motion differential equation of GCRS is established based on the finite element method. Moreover, the numerical calculation method of meshing force, considering the uniform distribution load on contact surface, is presented, and the mathematical predictive time−frequency characteristics are analyzed by the Newmark stepwise integral approach. Finally, a reduced-scale application of the propulsion shaft system is utilized to validate the effectiveness of the proposed dynamic model. For the sensibility to low-frequency vibration, the natural frequencies and vibration modes of GCRS are analyzed through the processing and analysis of acceleration signal. The experimental dynamic response and main components of vibration are respectively consistent with mathematical results, which demonstrate the effectiveness of the proposed dynamic model of GCRS with misalignment. Furthermore, it also shows that the proposed finite element analysis and calculation method are suitable for complex shafting, providing a novel thought for dynamic analysis of the propeller−shaft−hull coupled system of marine.