This study mainly investigated the nonlinear vibration performance of a rotor-casing coupling system containing a bolted flange connection. The dynamic equations of the coupling system were developed while considering the radial stiffness of the bolted flange structure, which contained a spigot, squirrel cage with ball bearing, and rotor-casing coupling vibration. To study the influence of the disk casing fixed-point rubbing fault on the coupling system’s nonlinear dynamic performance, an analytical model of the nonlinear impact forces was established, which considered the contact and vibration responses of the rotor and casing. The frictional force was obtained based on the Coulomb friction law. The iterative analysis of motion equations was performed utilizing the Newmark method. Then, the nonlinear dynamic behaviors of the coupled systems were examined using data, including a bifurcation diagram, spectrum plot, greatest Lyapunov exponents, etc. The effects of rubbing fault on the dynamic properties of system were investigated in detail, indicating that there were various motion states, which were described as periodic, multi-periodic, and quasi-periodic motions. Comparing the simulation results, it was found that rubbing fault seriously affected the motion stability of the rotor system. Finally, by gathering and examining the vibration data set from a test platform for rotor-casings with bolted joints, the correctness of the numerical simulation findings was confirmed. Additionally, the results of the experimental investigation agreed with that of the simulation. The dynamic distinguishing characteristics that were noticed can be used as an indicator for determining whether the fixed-point rubbing fault between the rotor and casing has become worse.