As an important over-current component of the waterjet propulsion system, the main function of a nozzle is to transform the mechanical energy of the propulsion pump into the kinetic energy of the water and eject the water flow to obtain thrust. In this study, the nozzle with different geometry and parameters was simulated based on computational fluid dynamics simulation and experiment. Numerical results show a good agreement with experimental results. The results show that the nozzle with a circular shape outlet shrinks evenly. Under the designed flow rate condition, the velocity uniformity of the circular nozzle is 0.26% and 0.34% higher than that of the elliptical nozzle and the rounded rectangle nozzle, respectively. The pump efficiency of the circular nozzle is 0.31% and 0.14% higher than that of the others. The pressure recovery and hydraulic loss of the circular nozzle are superior. The hydraulic characteristics of the propulsion pump and waterjet propulsion system are optimal when the nozzle area is 30% times the outlet area of the inlet duct. Thus, the shaft power, head, thrust, and system efficiency of the propulsion pump and waterjet propulsion system are maximized. The system efficiency curve decreases rapidly when the outlet area exceeds 30% times the outlet area of the inlet duct. The transition curve forms greatly affect thrust and system efficiency. The transition of the linear contraction shows improved uniformity, and the hydraulic loss is reduced. Furthermore, the hydraulic performance of the nozzle with a linear contraction transition is better than that of others.