Formation lithology identification is of great importance for reservoir characterization and petroleum exploration. Previous methods are based on cutting logging and well-logging data and have a significant time lag. In recent years, many machine learning methods have been applied to lithology identification by utilizing well-logging data, which may be affected by drilling fluid. Drilling string vibration data is a high-density ancillary data, and it has the advantages of low-latency, which can be acquired in real-time. Drilling string vibration data is more accessible and available compared to well-logging data in ultra-deep well drilling. Machine learning algorithms enable us to develop new lithology identification models based on these vibration data. In this study, a vibration dataset is used as the signal source, and the original vibration signal is filtered by Butterworth (BHPF). Vibration time−frequency characteristics were extracted into time−frequency images with the application of short-time Fourier transform (STFT). This paper develops lithology classification models using new data sources based on a convolutional neural network (CNN) combined with Mobilenet and ResNet. This model is used for complex formation lithology, including fine gravel sandstone, fine sandstone, and mudstone. This study also carries out related model accuracy verification and model prediction results interpretation. In order to improve the trustworthiness of decision-making results, the gradient-weighted class-activated thermal localization map is applied to interpret the results of the model. The final verification test shows that the single-sample decision time of the model is 10 ms, the test macro precision rate is 90.0%, and the macro recall rate is 89.3%. The lithology identification model based on vibration data is more efficient and accessible than others. In conclusion, the CNN model using drill string vibration supplies a superior method of lithology identification. This study provides low-latency lithology classification methods to ensure safe and fast drilling.