The quality of cotton seed is of great significance to the production of cotton in the cotton industry. In order to reduce the workload of the manual sorting of cotton seeds and improve the quality of cotton seed sorting, this paper proposed an image-detection method of cotton seed damage based on an improved YOLOv5 algorithm. Images of cotton seeds with different degrees of damage were collected in the same environment. Cotton seeds of three different damage degrees, namely, undamaged, slightly damaged, and seriously damaged, were selected as the research objects. Labeling software was used to mark the images of these cotton seeds and the marked images were input into the improved YOLOv5s detection algorithm for appearance-based damage identification. The algorithm added the lightweight upsampling operator CARAFE to the original YOLOv5s detection algorithm and also improved the loss function. The experimental results showed that the mAP_0.5 value of the improved algorithm reached 99.5% and the recall rate reached 99.3% when the uncoated cotton seeds were detected. When detecting coated cotton seeds, the mAP_0.5 value of the improved algorithm reached 99.2% and the recall rate reached 98.9%. Compared with the traditional appearance-based damage detection approach, the improved YOLOv5s proposed in this paper improved the recognition accuracy and processing speed, and exhibited a better adaptability and generalization ability. Therefore, the proposed method can provide a reference for the appearance detection of crop seeds.