To handle the gangue well and control the settlement of the surface, as well as to reduce the risk of water bleeding, settlement and even blockage and pipe breaking of the gangue slurry in the process of conveying, the rheological characteristics of the slurry should be studied. The rheological properties of slurry with different concentrations prepared from gangue samples of the Ningtiaota coal mine were tested, and the correlation between the rheological characteristics of the coal gangue filling slurry and three factors, namely the gangue mass fraction, grain gradation and standing time, were studied by a single factor method and response surface methodology. The results show that the fitting curve of the Herschel−Bulkley model is mostly linear, that is, the shear stress of coal gangue paste increases as a function of the shear rate. Therefore, these two concentrations are too small to form a stable network structure to wrap large particles and can easily cause pipe blockage. The yield shear stress and plastic viscosity show an exponential increase with the increasing mass fraction. The shear stress and apparent viscosity of the pastes with mass fractions of 60% and 65%, respectively, increase significantly after 20, 40 and 60 min of standing. According to the comprehensive test results and the response surface, the optimization method is as follows: mass fraction of 72%; aggregate grading for 4.75~1.18 mm particle size is 30%, for 1.18~0.425 mm particle size is 40%, for 0.425~0.075 mm particle size is 10%, for less than 0.075 mm particle size is 20%; with different standing times, the yield shear stress of slurry ranges from 103.02 to 131.645 Pa; and the plastic viscosity ranges from 0.54 to 0.64 Pa.s. With the increase of the standing time, the slurry settlement is relatively small, and is a more ideal gangue slurry proportion.