An algorithm based on the YOLOv5 model is proposed to address safety incidents such as tearing and blockage at transfer points on belt conveyors in coal mines caused by foreign objects mixed in with the coal flow. Given the tough underground conditions and images acquired with low quality, recursive filtering and MSRCR image enhancement algorithms were utilized to preprocess the dynamic images collected by underground monitoring devices, substantially enhancing image quality. The YOLOv5 model has been improved by introducing a multi-scale attention module (MSAM) during the channel map slicing, thereby increasing the model’s resistance to interference from redundant image features. Deep separable convolution was utilized in place of conventional convolution to detect, identify, and process large foreign objects on the belt conveyor as well as to increase detection speed. The MSAM-YOLOv5 model was trained before being installed on the NVIDIA Jetson Xavier NX platform and utilized to identify videos gathered from the coal mine belt conveyor. According to the experimental findings, the upgraded MSAM-YOLOv5 model has a greater recognition accuracy than YOLOv5L, with an average recall rate for different foreign objects of 96.27%, an average detection accuracy of 97.35%, and a recognition speed of 44 frames/s. The algorithm assures detection accuracy while increasing detection speed, satisfying the requirements for large foreign object detection on belt conveyors in coal mines.