A time−frequency residual convolution neural network (TFRCNN) was proposed to identify various rolling bearing fault types more efficiently. Three novel points about TFRCNN are presented as follows: First, by constructing a double-branch convolution network in the time domain and the frequency domain, the respective features in the time domain and the frequency domain were extracted to ensure the rich and complete feature representation of raw data sources. Second, specific residual structures were designed to prevent learning degradation of the deep network, and global average pooling was adopted to improve the network’s sparsity. Third, TFRCNN was better than the other models in terms of prediction accuracy, robustness, generalization ability, and convergence. The experimental results demonstrate that the prediction accuracy rate of TFRCNN, trained using mixing load data, reached 98.88 to 99.92% after optimizing the initial learning rate and choosing the optimizer and loss function. It was verified that TFRCNN can adaptively learn to extract deep fault features, accurately identify bearing fault conditions, and overcome the limitations of classical shallow feature extraction and classification methods, as well as common convolution neural networks. Hence, this investigation revealed TFRCNN’s potential for bearing fault diagnosis in practical engineering applications.